Tarea_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Competencia1.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {}
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iNaIFYN334UK",
        "colab_type": "text"
      },
      "source": [
        "# Tarea 1 NLP : Competencia de Clasificación de Texto\n",
        "-------------------------------\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tDaFlPl38xp",
        "colab_type": "text"
      },
      "source": [
        "- **Nombres:**\n",
        "\n",
        "\n",
        "\n",
        "> *   Mario Vicuña Álvarez\n",
        "> *   Miguel Videla Araya\n",
        "\n",
        "\n",
        "\n",
        "- **Usuario o nombre de equipo en Codalab:** \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfp7VkWN3-LZ",
        "colab_type": "text"
      },
      "source": [
        "## Objetivo e Instrucciones:\n",
        "\n",
        "### Objetivo\n",
        "\n",
        "Esta tarea consiste en participar en una competencia cuyo objetivo es la clasificación de tweets según su intensidad de emoción. Específicamente: \n",
        "\n",
        "Tendrán 4 datasets de tweets de distintas emociones: `anger`, `fear`, `sadness` y `joy`. Para cada uno de estos datasets, deberán crear un clasificador que indique la intensidad de dicha emoción en sus tweets (`low`, `medium`, `high`)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5AMxKnYs4CD5",
        "colab_type": "text"
      },
      "source": [
        "###  Fecha de Entrega: \n",
        "\n",
        "Por ser anunciada una vez termine el paro. Se publicará la fecha en ucursos."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_AEpGtid4IMw",
        "colab_type": "text"
      },
      "source": [
        "### Detalles e instrucciones de la competencia:\n",
        "\n",
        "- La competencia consiste en resolver 4 problemas de clasificación distintos, cada uno de tres clases. Por cada problema deberán crear un clasificador distinto. La evaluación de la competencia se realiza en base a 4 métricas: AUC, Kappa y Accuracy. Los mejores puntajes en cada ítem serán los que ganen.\n",
        "\n",
        "- Para comenzar se les entregará en este notebook el baseline y la estructura del reporte. El baseline es el código que realiza creación de features y clasificación básica. Los puntajes de este serán ocupados como base para la competencia: deben superar sus resultados para ser bien evaluados.  \n",
        "\n",
        "- Para participar, deben registrarse en Codalab y luego ingresar a la competencia usando el siguiente [link]( https://competitions.codalab.org/competitions/24121?secret_key=f5eb2d95-b36e-4aad-8fc5-4d9d77f4e4dc). \n",
        "\n",
        "- **Es requisito entregar el reporte con el código y haber participado en la competencia para ser evaluado.**\n",
        "\n",
        "- Pueden hacer grupos de máximo 2 alumnos. Cada grupo debe tener un nombre de equipo (En codalab, ir a settings y después cambiar Team Name). Solo una persona debe administrar la cuenta del grupo.\n",
        "\n",
        "- En total pueden hacer un **máximo de 4 envíos/submissions** (tanto para equipos como para envíos indivuales).\n",
        "\n",
        "- Hagan varios experimentos haciendo cross-validation o evaluación sobre una sub-partición antes de enviar sus predicciones a Codalab. Asegúrense que la distribución de las clases sea balanceada en las particiones de training y testing. Verificar que el formato de la submission coincida con el de la competencia. De lo contrario, se les será evaluado incorrectamente.\n",
        "\n",
        "- Estar top 5 en alguna métrica equivale a 1 punto extra en la nota final.\n",
        "\n",
        "- No se limiten a los contenidos vistos ni a scikit ni a este baseline. ¡Usen todo su conocimiento e ingenio en mejorar sus sistemas! \n",
        "\n",
        "- Todas las dudas escríbanlas en el hilo de U-cursos de la tarea. Los emails que lleguen al equipo docente serán remitidos a ese medio.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wASTl0OD4MYp",
        "colab_type": "text"
      },
      "source": [
        "### Reporte\n",
        "\n",
        "Este debe cumplir la siguiente estructura:\n",
        "\n",
        "1.\t**Introducción**: Presentar brevemente el problema a resolver, los métodos y representaciones utilizadas en el desarrollo de la tarea y conclusiones obtenidas. (0.5 Puntos)\n",
        "2.\t**Representaciones**: Describir los atributos y representaciones usadas como entrada de los clasificadores. Si bien, con Bag of Words (baseline) ya se comienzan a percibir buenos resultados, pueden mejorar su evaluación agregando más atributos y representaciones diseñadas a mano. Mas abajo encontrarán una lista útil de estos que les podrá ser de utilidad. (1.5 puntos)\n",
        "3.\t**Algoritmos**: Describir brevemente los algoritmos de clasificación usados. (0.5 puntos)\n",
        "4.\t**Métricas de evaluación**: Describir brevemente las métricas utilizadas en la evaluación indicando que miden y su interpretación. (0.5 puntos)\n",
        "5.\t**Experimentos**: Reportar todos sus experimentos. Comparar los resultados obtenidos utilizando diferentes algoritmos y representaciones. Estos experimentos los hacen sobre la sub-partición de evaluación que deben crear (o pueden usar cross-validation). Incluyan todo el código de sus experimentos aquí. ¡Es vital haber realizado varios experimentos para sacar una buena nota! (2 puntos)\n",
        "6.\t**Conclusiones**: Discutir resultados, proponer trabajo futuro. (1 punto)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iu1EEhNN4Q8Y",
        "colab_type": "text"
      },
      "source": [
        "### Baseline\n",
        "\n",
        "Por último, el baseline contiene un código básico que:\n",
        "\n",
        "- Obtiene los dataset.\n",
        "- Divide los datasets en train (entrenamiento y prueba) y target set (el que clasificar para subir a la competencia).\n",
        "- Crea un Pipeline que: \n",
        "    - Crea features personalizadas.\n",
        "    - Transforma los dataset a bag of words (BoW).  \n",
        "    - Entrena un clasificador usando cada train set.\n",
        "- Clasifica y evalua el sistema creado usando el test set.\n",
        "- Clasifica el target set.\n",
        "- Genera una submission con el target en formato zip en el directorio en donde se está ejecutando el notebook. \n",
        "\n",
        "\n",
        "Algunas pistas sobre como mejorar el rendimiento de los sistemas que creen. (Esto tendrá mas sentido cuando vean el código)\n",
        "\n",
        "- **Vectorizador**: investigar los modulos de `nltk`, en particular, `TweetTokenizer`, `mark_negation` para reemplazar los tokenizadores. También, el parámetro `ngram_range` (Ojo que el clf naive bayes no debería usarse con n-gramas, ya que rompe el supuesto de independencia). Además, implementar los atributos que crean útiles desde el listado del el enunciado. Investigar también el vectorizador tf-idf.\n",
        "\n",
        "- **Clasificador**: investigar otros clasificadores mas efectivos que naive bayes. Estos deben poder retornar la probabilidad de pertenecia de las clases (ie: implementar la función `predict_proba`).\n",
        "\n",
        "- **Features**: Recuerden que pueden implementar todas las features que se les ocurra! Aquí les adjuntamos algunos ejemplos:\n",
        "    -\tWord n-grams.\n",
        "    -\tCharacter n-grams. \n",
        "    -\tPart-of-speech tags.\n",
        "    -\tSentiment Lexicons (Lexicon = A set of words with a label or associated value.).\n",
        "        - Count the number of positive and negative words within a sentence.\n",
        "        - If the lexicon has associated intensity of feeling (for example in a decimal), then take the average of the intensity of the sentence according to the feeling, the sum, etc.\n",
        "        -\tA good lexicon of sentiment: [Bing Liu](http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar) \n",
        "        - A reference with a lot of [sentiment lexicons](https://medium.com/@datamonsters/sentiment-analysis-tools-overview-part-1-positive-and-negative-words-databases-ae35431a470c). \n",
        "    -\tThe number of elongated words (words with one character repeated more than two times).\n",
        "    -\tThe number of words with all characters in uppercase.\n",
        "    -\tThe presence and the number of positive or negative emoticons.\n",
        "    -\tThe number of individual negations.\n",
        "    -\tThe number of contiguous sequences of dots, question marks and exclamation marks.\n",
        "    -\tWord Embeddings: Here are some good ideas on how to use them.\n",
        "    https://stats.stackexchange.com/questions/221715/apply-word-embeddings-to-entire-document-to-get-a-feature-vector\n",
        "\n",
        "- **Reducción de dimensionalidad**: También puede serles de ayuda. Referencias [aquí](https://scikit-learn.org/stable/modules/unsupervised_reduction.html).\n",
        "\n",
        "- Por último, pueden encontrar mas referencias de cómo mejorar sus features, el vectorizador y el clasificador [aquí](https://affectivetweets.cms.waikato.ac.nz/benchmark/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kJc-aPbX4edh",
        "colab_type": "text"
      },
      "source": [
        "(Pueden eliminar cualquier celda con instrucciones...)\n",
        "\n",
        "**Importante**: Recuerden poner su nombre y el de su usuario o de equipo (en caso de que aplique) tanto en el reporte. NO serán evaluados Notebooks sin nombre."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HzsOVYqN4hXm",
        "colab_type": "text"
      },
      "source": [
        "----------------------------------------"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__zLuU0E4h8Z",
        "colab_type": "text"
      },
      "source": [
        "## 1. Introducción"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zfb9LpJa4jr5",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "37I_cTmB4mQY",
        "colab_type": "text"
      },
      "source": [
        "## 2. Representaciones"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LCe5onRf4oXD",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JuVjMZ8C4ojY",
        "colab_type": "text"
      },
      "source": [
        "## 3. Algoritmos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_A1TDhWd4qiH",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9jYeKgRM4qq4",
        "colab_type": "text"
      },
      "source": [
        "## 4. Métricas de Evaluación"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8mYjqlI4uUQ",
        "colab_type": "text"
      },
      "source": [
        "En el presente proyecto se consideran varias métricas de evaluación, las cuales en el contexto de la competencia pueden ser divididas en dos categorías según su uso.\n",
        "\n",
        "Por un lado se encuentran las métricas que no son directamente consideradas, en el sentido que en la competencia no serán usadas como un criterio de evaluación. No obstante, en el proceso de diseño y experimentación estas pueden ser consideradas para mejorar los algoritmos estudiados. Dentro de esta categoría se encuentran:\n",
        "\n",
        "- Precision: Corresponde a la fracción de clasificaciones correctas entre los datos clasificados como positivos. Matemáticamente Precision corresponde a:\n",
        "\n",
        "\\begin{equation}\n",
        "  P = \\frac{TP}{TP + FP}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ siendo $TP$ la cantidad de clasificaciones positivas correctas y $FP$ incorrectas. Conceptualmente, optimizar sobre la Precision corresponde a minimizar el Error de Tipo I, es decir que se busca una baja tasa de falsas detecciones.\n",
        "\n",
        "- Recall: Corresponde a la fracción de clasificaciones correctas entre los datos que verdaderamente corresponden a la clase positiva. Matemáticamente Recall corresponde a:\n",
        "\n",
        "\\begin{equation}\n",
        "  R = \\frac{TP}{TP + FN}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ siendo $FN$ los falsos negativos o datos de clase positiva incorrectamente clasificados. Conceptualmente, optimizar sobre el Recall $\\hspace{10mm}$ corresponde a minimizar el Error de Tipo II, es decir que se busca una baja tasa de no-detecciones.\n",
        "\n",
        "Típicamente existe un *trade-off* entre las métricas de Precision y Recall. Por  ejemplo, para maximizar el Recall un algoritmo podría determinar que todas sus entradas son postivas (con lo que $FN = 0$). Evidentemente con esto se dispara la tasa de falsos positivos, perjudicando evidentemente la Precision del modelo y, más aún, es un comportamiento que claramente no se desea en un clasificador.\n",
        "\n",
        "No obstante, se pueden combinar ambas métricas, por ejemplo mediante el F1-Score.\n",
        "\n",
        "- F1-Score: El F1-Score funge como una métrica que combina ambas con la finalidad de aproximarse a un punto óptimo, por medio de una media harmónica ponderada mediante un parámetro de balance $\\alpha$. Esta métrica, en términos de $P$ y $R$ se define matemáticamente como:\n",
        "\n",
        "\\begin{equation}\n",
        "  F_{\\alpha} = \\frac{1}{\\alpha \\frac{1}{P} + (1-\\alpha) \\frac{1}{R}}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ También se puede escribir esta expresión en términos del factor de balance $\\beta$ de la siguiente forma:\n",
        "\n",
        "\\begin{equation}\n",
        "  F_{\\beta} = (1 + \\beta^{2}) \\frac{PR}{\\beta^{2}P + R}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ Cuando $\\beta = 1$, o equivalentemente $\\alpha = \\frac{1}{2}$, $F_{\\beta} = F_{1}$ toma precisamente la forma de la media harmónica (balanceada) entre $P$ y $R$, de ahí que esta métrica se conozca como F1-Score. En términos generales se la denomina simplemente F-Score, pudiendo considerar cualquier valor para $\\alpha$ o $\\beta$.\n",
        "\n",
        "Por otra parte, entre las métricas que efectivamente serán consideradas para medir el desempeño de cada participante en la competencia son:\n",
        "\n",
        "- Accuracy: Esta es otra métrica \"clásica\" para algoritmos de clasificación, y corresponde a la razón de clasificaciones correctas (tanto positivas como negativas). En este sentido, se busca que el algoritmo acierte tanto al discriminar tanto los casos positivos como los negativos, por lo que la métrica a maximizar es:\n",
        "\n",
        "\\begin{equation}\n",
        "  Acc = \\frac{TP + TN}{TP + FP + TN + FN}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ Planteado de esta forma pareciera que es evidente que esta métrica cumple con lo que intuitivamente se esperaría de un sistema de clasificación/detección, al minimizar las equivocaciones **en general**, tanto para los casos de clase positiva como negativa. No obstante para datos desbalanceados podría incurrirse en errores como el dado de ejemplo para el caso del Recall. Por ejemplo, si los casos de clase positiva son escasos, un algoritmo de clasificación puede lograr altos valores de Accuracy catalogando todos los casos como negativos (la que sería la clase mayoritaria). Este tipo de análisis y consideraciones toma especial relevancia al momento de tratar problemas de varias clases o de la vida real, como diagnósticos médicos.\n",
        "\n",
        "\n",
        "- AUC: Para entender la métrica de AUC (*Area under Curve*) es necesario introducir primero el concepto de Curva ROC. A grandes rasgos, la curva ROC captura el rendiento del modelo en distintos regímenes o umbrales entre la tasa de falsos positivos y la tasa de verdaderos positivos, como se ve en la siguiente figura ilustrativa:\n",
        "\n",
        "![texto alternativo](https://drive.google.com/uc?export=view&id=1XdlcSB9nUiRMy0BtVivbsBOHU5UGA0B8)\n",
        "\n",
        "$\\hspace{10mm}$ Un regimen se define a partir del umbral por el cual se toma una decisión, por ejemplo si la probabilidad de pertener a la clase posiva es mayor a dicho umbral, se califica como perteneciente a dicha clase y como negativa en caso contrario. Por lo tanto, al cambiar dicho umbral se alteran las capacidades de distinguir una clase de otra. No obstante, la forma de la curva no depende solo del umbral de decisión; de hecho para un umbral dado solo se tiene un punto de la curva y al cambiar dicho umbral dicho punto se traslada **a través** de la curva. \n",
        "\n",
        "$\\hspace{10mm}$ La forma de la curva ROC está dada por el test de hipótesis que se desea modelar. Para ilustrar esto sean dos distribuciones de probabilidad sobre el espacio de observaciones. Al muestrear un valor en dicho espacio, el propósito del problema de detección es determinar de qué distribución proviene dicha muestra. Naturalmente, cuando las distribuciones se solapan de manera leve el problema es sencillo puesto que en términos prácticos poseen soportes disjuntos. Por otro lado, si las distribuciones correspondientes a la clase positiva y negativa coinciden, el problema se hace indistinguible, pues una muestra dada es igualmente verosimil en ambos casos. Considerando estos dos ejemplos es posible inferir la forma que tiene la curva ROC según dicho traslape e interpretar a partir de los ejes: \n",
        "\n",
        "> Cuando el problema es indistinguible (las distribuciones coinciden) trazar un umbral de decisión hace que se acepten como positivas tantas muestras positivas como negativas, por lo que existe una relación uno a uno entre la tasa de verdaderos positivos y la tasa de falsos positivos. En consecuencia, la curva ROC es diagonal de pendiente unitaria y los puntos sobre esta son de la forma $(x,x), x \\in [0,1]$.\n",
        "\n",
        "> Cuando el problema es perfectamente separable, es decir que las distribuciones de ambas clases presentan muy poco traslape, un umbral de decisión permite rechazar la gran mayoría de muestras pertenecientes a la clase negativa, sin perjudicar la correcta detección de las de clase positiva. En tal caso, la curva ROC aproxima un escalón unitario con dominio en $[0,1]$.\n",
        "\n",
        "$\\hspace{10mm}$ [Acá](http://www.navan.name/roc/) se presenta una GUI *online* que muestra la curva ROC correspondiente dadas dos distribuciones Gaussianas para las clases y un umbral, donde se aprecian los comportamientos descritos.\n",
        "\n",
        "$\\hspace{10mm}$ En consecuencia, los mejores desempeños son obtenibles en los casos en que la curva ROC se aproxima a la función escalón. De aquí el interés de la métrica AUC: la curva ROC está \"acotada\" por el escalón cuya área es $1$, por lo tanto a mayor área bajo la curva ROC, más similar es esta a su cota superior y es posible hacer detecciones más precisas.\n",
        "\n",
        "$\\hspace{10mm}$ Planteado de esta forma, esta es una métrica que depende solamente del problema estudiado, mientras que la dependencia con el sistema de decisión es muy poco sustancial. No obstante, muchos problemas en la práctica no cuentan con un modelo estadístico como el planteado, es decir que no se conoce ninguna de las dos distribuciones. Acá es justamente donde entran en juego los modelos de aprendizaje, pues estos \"aprenden las distribuciones\" a partir de los datos y en consecuencia es el propio algoritmo el que define la curva ROC al reconocer las distribuciones. Luego, el criterio para elegir un modelo basado en esta métrica es aquel que tras ser entrenado obtenga la mayor área bajo la curva (la cual se obtiene en forma numérica).\n",
        "\n",
        "- Kappa: En términos generales, el Coeficiente Kappa ($\\kappa$) de Cohen mide estadísticamente el grado de acuerdo entre dos examinadores/clasificadores, considerando el efecto del azar. En este sentido, se penaliza la probabilidad de que ambos clasificadores hayan determinado el mismo resultado por mero efecto estadístico más que por algún esquema de decisión subyacente. Para ilustrar esto analicemos la definición de esta métrica:\n",
        "\n",
        "\\begin{equation}\n",
        " \\kappa = \\frac{\\mathbb{P}(a) - \\mathbb{P}(r)}{1 - \\mathbb{P}(r)}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ donde el evento $a$ corresponde al acuerdo entre los examinadores y el evento $r$ al acuerdo puramente por azar. Por propiedades de la medida de probabilidad, el numerador de $\\kappa$ corresponde efectivamente a la probabilidad de que ambos clasificadores lleguen a una misma conclusión \"sin recurrir\" al azar.\n",
        "\n",
        "$\\hspace{10mm}$ Si se considera que uno de estos examinadores es precisamente la *ground truth* presente en los datos este análisis se puede enriquecer aún mas. En términos de objetivos, se espera que un clasificador se comporte en la forma en que lo hace el \"oráculo\" que proporciona la *ground truth*, es decir que $\\mathbb{P}(a) = 1$, en cuyo caso $\\kappa = 1$. Por otro lado, también se espera que nuestro modelo tome decisiones en forma \"inteligente\" por lo que se busca reducir $\\mathbb{P}(r)$, evidentemente hasta cero es el ideal. En cualquier caso el valor óptimo de esta métrica es $1$, en el cual la matriz de confusión es diagonal.\n",
        "\n",
        "$\\hspace{10mm}$ Más aún, cabe considerar que tanto $\\mathbb{P}(a)$ como $\\mathbb{P}(r)$ se obtienen en forma empírica. Luego, si se considera que uno de los examinadores proporciona las clases reales de los datos, es posible escribir ambas probabilidades estimadas en términos de $TP, FP, TN$ y $FN$:\n",
        "\n",
        "$\\hspace{10mm}$ Claramente, $\\mathbb{P}(a)$ corresponde a la Accuracy del modelo, pues corresponde a la proporción de aciertos, es decir:\n",
        "\n",
        "\\begin{equation}\n",
        "  \\mathbb{P}(a) = Acc = \\frac{TP + TN}{TP + FP + TN + FN}\n",
        "\\end{equation}\n",
        "\n",
        "$\\hspace{10mm}$ Por otro lado, típicamente la probabilidad $\\mathbb{P}(r)$ se modela como:\n",
        "\n",
        "\\begin{equation}\n",
        "  \\mathbb{P}(r) = \\mathbb{P}_{A}(Sí)*\\mathbb{P}_{B}(Sí) + \\mathbb{P}_{A}(No)*\\mathbb{P}_{B}(No)\n",
        "\\end{equation}\n",
        "\n",
        "donde $\\mathbb{P}{i}(X)$ denota la probabilidad de que el examinador $i$ determine $X$. Cuando el examinador $B$ corresponde a la *ground truth* sus probabilidades estimadas corresponden a:\n",
        "\n",
        "\\begin{equation}\n",
        "  \\mathbb{P}_{B}(Sí) = \\frac{TP}{TP + TN}, \\hspace{5mm} \\mathbb{P}_{B}(No) = \\frac{TN}{TP + TN}\n",
        "\\end{equation}\n",
        "\n",
        "> Ojo: esto corresponde nada más que a la distribución de clases en los datos. Consecuencia de esto, la métrica dada por $\\kappa$ incorpora naturalmente una medida del desbalance en los datos.\n",
        "\n",
        "mientras que para el examinador $A$ (correspondiente al modelo evaluado):\n",
        "\n",
        "\\begin{equation}\n",
        "  \\mathbb{P}_{A}(Sí) = \\frac{TP + FP}{TP + FP + TN + FN}, \\hspace{5mm} \\mathbb{P}_{A}(No) = \\frac{TN + FN}{TP + FP + TN + FN}\n",
        "\\end{equation}\n",
        "\n",
        "```python\n",
        "Classification Report:\n",
        "\n",
        "              precision    recall  f1-score   support\n",
        "\n",
        "         low       0.46      0.12      0.19        49\n",
        "      medium       0.67      0.92      0.77       204\n",
        "        high       0.44      0.14      0.21        58\n",
        "    accuracy                           0.65       311\n",
        "   macro avg       0.52      0.39      0.39       311\n",
        "weighted avg       0.59      0.65      0.58       311\n",
        "\n",
        "Scores:\n",
        "\n",
        "AUC:  0.611\tKappa: 0.098\tAccuracy: 0.646\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NBsrTL6N4u-3",
        "colab_type": "text"
      },
      "source": [
        "## 5. Experimentos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xX-I5gMZF8ZK",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJQJ0rlz42bx",
        "colab_type": "text"
      },
      "source": [
        "### Importar librerías y utiles"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "roAVXXYl4_j5",
        "colab_type": "code",
        "outputId": "4bb45a8d-3db7-43ae-f685-df38e384bddf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 530
        }
      },
      "source": [
        "!pip install emoji\n",
        "!pip install --upgrade ipykernel -q\n",
        "!sudo apt install openjdk-8-jdk\n",
        "!sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java\n",
        "!pip install allennlp -q\n",
        "!pip install transformers -q\n",
        "#!pip install transformers==2.8.0\n",
        "!pip install pycontractions -q\n",
        "\n",
        "import os\n",
        "import re\n",
        "import time\n",
        "import torch\n",
        "import emoji\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import shutil\n",
        "\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.ensemble import RandomForestClassifier as RFC\n",
        "from sklearn.ensemble import VotingClassifier\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.neighbors import KNeighborsClassifier as KNC\n",
        "from sklearn.neural_network import MLPClassifier as MLP\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.pipeline import Pipeline, FeatureUnion\n",
        "from sklearn.metrics import confusion_matrix, cohen_kappa_score, classification_report, accuracy_score, roc_auc_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.base import BaseEstimator, TransformerMixin\n",
        "from sklearn.preprocessing import StandardScaler"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: emoji in /usr/local/lib/python3.6/dist-packages (0.5.4)\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "openjdk-8-jdk is already the newest version (8u252-b09-1~18.04).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded.\n",
            "Requirement already satisfied: transformers==2.8.0 in /usr/local/lib/python3.6/dist-packages (2.8.0)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (0.1.86)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (4.41.1)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (1.13.4)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (1.18.4)\n",
            "Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (0.0.43)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (0.7)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (2.23.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (3.0.12)\n",
            "Requirement already satisfied: tokenizers==0.5.2 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (0.5.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.8.0) (2019.12.20)\n",
            "Requirement already satisfied: botocore<1.17.0,>=1.16.4 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.8.0) (1.16.4)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.8.0) (0.9.5)\n",
            "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers==2.8.0) (0.3.3)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (0.14.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.8.0) (1.12.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2020.4.5.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (2.9)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.8.0) (3.0.4)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.4->boto3->transformers==2.8.0) (0.15.2)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.17.0,>=1.16.4->boto3->transformers==2.8.0) (2.8.1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q9QjH0YleVaC",
        "colab_type": "code",
        "outputId": "ba1cee15-09d5-46f1-8862-f64991d720ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import transformers\n",
        "print(transformers.__version__)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.8.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vuURQYHN5SzP",
        "colab_type": "text"
      },
      "source": [
        "### Definir métodos de evaluación\n",
        "\n",
        "Estas funciones están a cargo de evaluar los resultados de la tarea. No deberían cambiarlas.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x2ceYCue5Uch",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def auc_score(test_set, predicted_set):\n",
        "    high_predicted = np.array([prediction[2] for prediction in predicted_set])\n",
        "    medium_predicted = np.array(\n",
        "        [prediction[1] for prediction in predicted_set])\n",
        "    low_predicted = np.array([prediction[0] for prediction in predicted_set])\n",
        "    high_test = np.where(test_set == 'high', 1.0, 0.0)\n",
        "    medium_test = np.where(test_set == 'medium', 1.0, 0.0)\n",
        "    low_test = np.where(test_set == 'low', 1.0, 0.0)\n",
        "    auc_high = roc_auc_score(high_test, high_predicted)\n",
        "    auc_med = roc_auc_score(medium_test, medium_predicted)\n",
        "    auc_low = roc_auc_score(low_test, low_predicted)\n",
        "    auc_w = (low_test.sum() * auc_low + medium_test.sum() * auc_med +\n",
        "             high_test.sum() * auc_high) / (\n",
        "                 low_test.sum() + medium_test.sum() + high_test.sum())\n",
        "    return auc_w\n",
        "\n",
        "\n",
        "def evaulate(predicted_probabilities, y_test, labels, dataset_name):\n",
        "    # Importante: al transformar los arreglos de probabilidad a clases,\n",
        "    # entregar el arreglo de clases aprendido por el clasificador.\n",
        "    # (que comunmente, es distinto a ['low', 'medium', 'high'])\n",
        "    predicted_labels = [\n",
        "        labels[np.argmax(item)] for item in predicted_probabilities\n",
        "    ]\n",
        "    print('Confusion Matrix for {}:\\n'.format(dataset_name))\n",
        "    print(\n",
        "        confusion_matrix(y_test,\n",
        "                         predicted_labels,\n",
        "                         labels=['low', 'medium', 'high']))\n",
        "\n",
        "    print('\\nClassification Report:\\n')\n",
        "    print(\n",
        "        classification_report(y_test,\n",
        "                              predicted_labels,\n",
        "                              labels=['low', 'medium', 'high']))\n",
        "    # Reorder predicted probabilities array.\n",
        "    labels = labels.tolist()\n",
        "    predicted_probabilities = predicted_probabilities[:, [\n",
        "        labels.index('low'),\n",
        "        labels.index('medium'),\n",
        "        labels.index('high')\n",
        "    ]]\n",
        "    auc = round(auc_score(y_test, predicted_probabilities), 3)\n",
        "    print(\"Scores:\\n\\nAUC: \", auc, end='\\t')\n",
        "    kappa = round(cohen_kappa_score(y_test, predicted_labels), 3)\n",
        "    print(\"Kappa:\", kappa, end='\\t')\n",
        "    accuracy = round(accuracy_score(y_test, predicted_labels), 3)\n",
        "    print(\"Accuracy:\", accuracy)\n",
        "    print('------------------------------------------------------\\n')\n",
        "    return np.array([auc, kappa, accuracy])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1F6AgPd-5bR5",
        "colab_type": "text"
      },
      "source": [
        "### Datos\n",
        "\n",
        "Obtener los datasets desde el github del curso"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMRyT9SA5dvI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Datasets de entrenamiento.\n",
        "train = {\n",
        "    'anger': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/train/anger-train.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity']),\n",
        "    'fear': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/train/fear-train.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity']),\n",
        "    'joy': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/train/joy-train.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity']),\n",
        "    'sadness': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/train/sadness-train.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity'])\n",
        "}\n",
        "# Datasets que deberán predecir para la competencia.\n",
        "# ie target ------> test set\n",
        "target = {\n",
        "    'anger': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/target/anger-target.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity'], na_values=['NONE']),\n",
        "    'fear': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/target/fear-target.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity'], na_values=['NONE']),\n",
        "    'joy': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/target/joy-target.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity'], na_values=['NONE']),\n",
        "    'sadness': pd.read_csv('https://raw.githubusercontent.com/dccuchile/CC6205/master/assignments/assignment_1/data/target/sadness-target.txt', sep='\\t', names=['id', 'tweet', 'class', 'sentiment_intensity'], na_values=['NONE'])\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mLKwQxGZ5giQ",
        "colab_type": "code",
        "outputId": "b9521b1c-4578-47df-f6ee-fdf828c9ddd5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Ejemplo de algunas filas aleatorias:\n",
        "train['anger'].sample(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>tweet</th>\n",
              "      <th>class</th>\n",
              "      <th>sentiment_intensity</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>77</th>\n",
              "      <td>10077</td>\n",
              "      <td>What the fuck am I supposed to do with no lunc...</td>\n",
              "      <td>anger</td>\n",
              "      <td>high</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>601</th>\n",
              "      <td>10601</td>\n",
              "      <td>straight people are canoodling on the quad and...</td>\n",
              "      <td>anger</td>\n",
              "      <td>medium</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>110</th>\n",
              "      <td>10110</td>\n",
              "      <td>@jennylhowe I am angry at the student for bein...</td>\n",
              "      <td>anger</td>\n",
              "      <td>high</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>235</th>\n",
              "      <td>10235</td>\n",
              "      <td>@DFSCare apparently you are to contact me. Sof...</td>\n",
              "      <td>anger</td>\n",
              "      <td>medium</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>676</th>\n",
              "      <td>10676</td>\n",
              "      <td>Nurse practitioner: 'you look pretty bad. No o...</td>\n",
              "      <td>anger</td>\n",
              "      <td>medium</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id  ... sentiment_intensity\n",
              "77   10077  ...                high\n",
              "601  10601  ...              medium\n",
              "110  10110  ...                high\n",
              "235  10235  ...              medium\n",
              "676  10676  ...              medium\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CDSBDKCC5ith",
        "colab_type": "text"
      },
      "source": [
        "### Analizar los datos \n",
        "\n",
        "Imprimir la cantidad de tweets de cada dataset, según su intensidad de sentimiento. Noten que las clases están desbalanceadas. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j-L2PN2Z5ks4",
        "colab_type": "code",
        "outputId": "0f04b88c-7f4e-4ede-e004-961e676ea081",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561
        }
      },
      "source": [
        "def get_group_dist(group_name, train):\n",
        "    print(group_name, \"\\n\",\n",
        "          train[group_name].groupby('sentiment_intensity').count(),\n",
        "          '\\n---------------------------------------\\n')\n",
        "for dataset_name in train:\n",
        "    get_group_dist(dataset_name, train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "anger \n",
            "                       id  tweet  class\n",
            "sentiment_intensity                   \n",
            "high                 163    163    163\n",
            "low                  161    161    161\n",
            "medium               617    617    617 \n",
            "---------------------------------------\n",
            "\n",
            "fear \n",
            "                       id  tweet  class\n",
            "sentiment_intensity                   \n",
            "high                 270    270    270\n",
            "low                  288    288    288\n",
            "medium               699    699    699 \n",
            "---------------------------------------\n",
            "\n",
            "joy \n",
            "                       id  tweet  class\n",
            "sentiment_intensity                   \n",
            "high                 195    195    195\n",
            "low                  219    219    219\n",
            "medium               488    488    488 \n",
            "---------------------------------------\n",
            "\n",
            "sadness \n",
            "                       id  tweet  class\n",
            "sentiment_intensity                   \n",
            "high                 197    197    197\n",
            "low                  210    210    210\n",
            "medium               453    453    453 \n",
            "---------------------------------------\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nqZ80fGY6BlY",
        "colab_type": "text"
      },
      "source": [
        "### 5.1. Baseline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S4TZ72FfGlj2",
        "colab_type": "text"
      },
      "source": [
        "A continuación se presenta el baseline en base al cual se realizarán las comparaciones pertinentes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "89bUbdWF5nMw",
        "colab_type": "text"
      },
      "source": [
        "#### Custom Features \n",
        "\n",
        "Para crear features personalizadas implementaremos nuestros propios Transformers (estandar de scikit para crear nuevas features entre otras cosas). Para esto:\n",
        "\n",
        "1. Creamos nuestra clase Transformer extendiendo BaseEstimator y TransformerMixin. En este ejemplo, definiremos `CharsCountTransformer` que cuenta carácteres relevantes ('!', '?', '#') en los tweets.\n",
        "2. Definios una función cómo `get_relevant_chars` que opera por cada tweet y retorna un arreglo.\n",
        "3. Hacemos un override de la función `transform` en donde iteramos por cada tweet, llamamos a la función que hicimos antes y agregamos sus resultados a un arrelo. Finalmente lo retornamos.\n",
        "\n",
        "Esto nos facilitará el trabajo mas adelante. Una Guia completa de las transformaciones predefinidas en scikit pueden encontrarla [aquí](https://scikit-learn.org/stable/data_transforms.html).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iI6y7r2a5pjm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CharsCountTransformer(BaseEstimator, TransformerMixin):\n",
        "    def get_relevant_chars(self, tweet):\n",
        "        num_hashtags = tweet.count('#')\n",
        "        num_exclamations = tweet.count('!')\n",
        "        num_interrogations = tweet.count('?')\n",
        "        return [num_hashtags, num_exclamations, num_interrogations]\n",
        "\n",
        "    def transform(self, X, y=None):\n",
        "        chars = []\n",
        "        for tweet in X:\n",
        "            chars.append(self.get_relevant_chars(tweet))\n",
        "\n",
        "        return np.array(chars)\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "        return self"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3RkxWZ0d5rKt",
        "colab_type": "code",
        "outputId": "dc9be64d-beab-434c-ed2c-6c9f04b73ea8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        }
      },
      "source": [
        "# Veamos que sucede si ejecutamos el transformer\n",
        "sample = train['anger'].sample(5).tweet\n",
        "sample = train['anger'].tweet\n",
        "pd.DataFrame(zip(sample, CharsCountTransformer().transform(sample)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>How the fu*k! Who the heck! moved my fridge!.....</td>\n",
              "      <td>[4, 3, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>So my Indian Uber driver just called someone t...</td>\n",
              "      <td>[1, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@DPD_UK I asked for my parcel to be delivered ...</td>\n",
              "      <td>[2, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>so ef whichever butt wipe pulled the fire alar...</td>\n",
              "      <td>[13, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Don't join @BTCare they put the phone down on ...</td>\n",
              "      <td>[1, 1, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>936</th>\n",
              "      <td>@Jen_ny69 People will always get offended ever...</td>\n",
              "      <td>[0, 1, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>937</th>\n",
              "      <td>@gayla_weeks1 I try not to let my anger seep i...</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>938</th>\n",
              "      <td>I hope my hustle don't offend nobody</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>939</th>\n",
              "      <td>Just watched Django Unchained, Other people ma...</td>\n",
              "      <td>[0, 1, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>940</th>\n",
              "      <td>Lol little things like that make me so angry x</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>941 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                     0           1\n",
              "0    How the fu*k! Who the heck! moved my fridge!.....   [4, 3, 0]\n",
              "1    So my Indian Uber driver just called someone t...   [1, 0, 0]\n",
              "2    @DPD_UK I asked for my parcel to be delivered ...   [2, 0, 0]\n",
              "3    so ef whichever butt wipe pulled the fire alar...  [13, 0, 0]\n",
              "4    Don't join @BTCare they put the phone down on ...   [1, 1, 0]\n",
              "..                                                 ...         ...\n",
              "936  @Jen_ny69 People will always get offended ever...   [0, 1, 0]\n",
              "937  @gayla_weeks1 I try not to let my anger seep i...   [0, 0, 0]\n",
              "938               I hope my hustle don't offend nobody   [0, 0, 0]\n",
              "939  Just watched Django Unchained, Other people ma...   [0, 1, 0]\n",
              "940     Lol little things like that make me so angry x   [0, 0, 0]\n",
              "\n",
              "[941 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D0Xtcs7U6P_Q",
        "colab_type": "text"
      },
      "source": [
        "#### Definir la representación y el clasificador\n",
        "\n",
        "Para esto, definiremos Pipelines. Un `Pipeline` es una lista de transformaciones y un estimador(clasificador) ubicado al final el cual define el flujo que seguiran nuestros datos dentro del sistema que creemos. Nos permite ejecutar facilmente el mismo proceso sobre todos los datasets que usemos, simplificando así nuestra programación.\n",
        "\n",
        "El pipeline más básico que podemos hacer es transformar el dataset a Bag of Words y después usar clasificar el BoW usando NaiveBayes:\n",
        "\n",
        "```python\n",
        "    Pipeline([('bow', CountVectorizer()), ('clf', MultinomialNB())])\n",
        "```\n",
        "\n",
        "\n",
        "Ahora, si queremos usar nuestra transformación para agregar las features que creamos, usaremos `FeatureUnion`. Esta simplemente concatenará los vectores resultantes de ejecutar BoW y los Transformer en un solo vector.\n",
        "\n",
        "```python\n",
        "    Pipeline([('features',FeatureUnion([('bow', CountVectorizer()),\n",
        "                                        ('chars_count',CharsCountTransformer())])),\n",
        "              ('clf', MultinomialNB())])\n",
        "\n",
        "```\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vgdoHa46UUp",
        "colab_type": "text"
      },
      "source": [
        "Recuerden que cada pipeline representa un sistema de clasificación distinto. Por lo mismo, deben instanciar uno por cada problema que resuelvan. De lo contrario, podrían solapar resultados.  Para esto, les recomendamos crear los pipeline en distintas funciones, como la siguiente:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jcYP5Puo6VQ3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_0_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fzVPf42D6cyP",
        "colab_type": "text"
      },
      "source": [
        "#### Ejecutar el pipeline para algún dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vS0sZpq56eBv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def run(dataset, dataset_name, pipeline):\n",
        "    \"\"\"Creamos el pipeline y luego lo ejecutamos el pipeline sobre un dataset. \n",
        "    Retorna el modelo ya entrenado mas sus labels asociadas y los scores obtenidos al evaluarlo.\"\"\"\n",
        "\n",
        "    # Dividimos el dataset en train y test.\n",
        "    X_train, X_test, y_train, y_test = train_test_split(\n",
        "        dataset.tweet,\n",
        "        dataset.sentiment_intensity,\n",
        "        random_state = 1138,\n",
        "        shuffle=True,\n",
        "        test_size=0.33)\n",
        "\n",
        "    # Entrenamos el clasificador (Ejecuta el entrenamiento sobre todo el pipeline)\n",
        "    pipeline.fit(X_train, y_train)\n",
        "\n",
        "    # Predecimos las probabilidades de intensidad de cada elemento del set de prueba.\n",
        "    predicted_probabilities = pipeline.predict_proba(X_test)\n",
        "\n",
        "    # Obtenemos el orden de las clases aprendidas.\n",
        "    learned_labels = pipeline.classes_\n",
        "\n",
        "    # Evaluamos:\n",
        "    scores = evaulate(predicted_probabilities, y_test, learned_labels, dataset_name)\n",
        "    return pipeline, learned_labels, scores"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhLqyi876gh5",
        "colab_type": "text"
      },
      "source": [
        "#### Ejecutar el sistema creado por cada train set\n",
        "\n",
        "Este código crea y entrena los 4 sistemas de clasificación y luego los evalua. Para los experimentos, pueden copiar este código variando el pipeline cuantas veces estimen conveniente."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hCyvG4wAArim",
        "colab_type": "code",
        "outputId": "52aaf0e6-701d-4e84-dc31-7b76f3216d9d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_0_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  50   0]\n",
            " [  1 188  14]\n",
            " [  0  44   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.86      0.11      0.19        56\n",
            "      medium       0.67      0.93      0.78       203\n",
            "        high       0.36      0.15      0.22        52\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.63      0.40      0.39       311\n",
            "weighted avg       0.65      0.65      0.58       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.576\tKappa: 0.106\tAccuracy: 0.65\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  73   4]\n",
            " [ 16 200  17]\n",
            " [  2  56  29]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.19      0.27        95\n",
            "      medium       0.61      0.86      0.71       233\n",
            "        high       0.58      0.33      0.42        87\n",
            "\n",
            "    accuracy                           0.60       415\n",
            "   macro avg       0.56      0.46      0.47       415\n",
            "weighted avg       0.58      0.60      0.55       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.676\tKappa: 0.206\tAccuracy: 0.595\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 16  50   0]\n",
            " [ 13 136  15]\n",
            " [  2  44  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.24      0.33        66\n",
            "      medium       0.59      0.83      0.69       164\n",
            "        high       0.59      0.32      0.42        68\n",
            "\n",
            "    accuracy                           0.58       298\n",
            "   macro avg       0.57      0.47      0.48       298\n",
            "weighted avg       0.58      0.58      0.55       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.677\tKappa: 0.206\tAccuracy: 0.584\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  52   1]\n",
            " [  7 137  11]\n",
            " [  1  46  19]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.16      0.25        63\n",
            "      medium       0.58      0.88      0.70       155\n",
            "        high       0.61      0.29      0.39        66\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.58      0.44      0.45       284\n",
            "weighted avg       0.58      0.58      0.53       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.64\tKappa: 0.184\tAccuracy: 0.585\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.642\t Average Kappa: 0.175\t Average Accuracy: 0.604\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7NldqaMkc3eP",
        "colab_type": "text"
      },
      "source": [
        "Notar que en la función `run` se incorpora una semilla a la acción de shuffling para particionar los datos para poder hacer comparaciones válidas entre los distintos modelos, al ser estos entrenados sobre la misma partición de los datos.\n",
        "\n",
        "Sin embargo, no se ha considerado el desbalance de clases. Para esto se incorpora en la función `train_test_split` el parámetro de `stratify`. A continuación se muestra el efecto de esto sobre el desempeño del modelo:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Na2-6tsHdqnf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def run(dataset, dataset_name, pipeline):\n",
        "    \"\"\"Creamos el pipeline y luego lo ejecutamos el pipeline sobre un dataset. \n",
        "    Retorna el modelo ya entrenado mas sus labels asociadas y los scores obtenidos al evaluarlo.\"\"\"\n",
        "\n",
        "    # Dividimos el dataset en train y test.\n",
        "    X_train, X_test, y_train, y_test = train_test_split(\n",
        "        dataset.tweet,\n",
        "        dataset.sentiment_intensity,\n",
        "        random_state = 1138,\n",
        "        shuffle=True,\n",
        "        test_size=0.33,\n",
        "        stratify=dataset.sentiment_intensity)\n",
        "\n",
        "    # Entrenamos el clasificador (Ejecuta el entrenamiento sobre todo el pipeline)\n",
        "    pipeline.fit(X_train, y_train)\n",
        "\n",
        "    # Predecimos las probabilidades de intensidad de cada elemento del set de prueba.\n",
        "    predicted_probabilities = pipeline.predict_proba(X_test)\n",
        "\n",
        "    # Obtenemos el orden de las clases aprendidas.\n",
        "    learned_labels = pipeline.classes_\n",
        "\n",
        "    # Evaluamos:\n",
        "    scores = evaulate(predicted_probabilities, y_test, learned_labels, dataset_name)\n",
        "    return pipeline, learned_labels, scores"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MJEBtgord2Cw",
        "colab_type": "code",
        "outputId": "a439e037-6952-415d-a320-ac08833fe698",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_0_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 188  11]\n",
            " [  1  38  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.11      0.18        53\n",
            "      medium       0.69      0.92      0.79       204\n",
            "        high       0.58      0.28      0.38        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.59      0.44      0.45       311\n",
            "weighted avg       0.64      0.67      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.654\tKappa: 0.186\tAccuracy: 0.672\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 17  75   3]\n",
            " [ 14 205  12]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.18      0.27        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.57      0.22      0.32        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.56      0.43      0.43       415\n",
            "weighted avg       0.57      0.58      0.52       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.646\tKappa: 0.162\tAccuracy: 0.583\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 17  52   3]\n",
            " [ 14 136  11]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.24      0.32        72\n",
            "      medium       0.61      0.84      0.71       161\n",
            "        high       0.66      0.42      0.51        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.50      0.51       298\n",
            "weighted avg       0.60      0.60      0.57       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.722\tKappa: 0.263\tAccuracy: 0.604\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  9  57   3]\n",
            " [  5 134  11]\n",
            " [  2  49  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.13      0.21        69\n",
            "      medium       0.56      0.89      0.69       150\n",
            "        high       0.50      0.22      0.30        65\n",
            "\n",
            "    accuracy                           0.55       284\n",
            "   macro avg       0.54      0.41      0.40       284\n",
            "weighted avg       0.55      0.55      0.48       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.654\tKappa: 0.136\tAccuracy: 0.553\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.669\t Average Kappa: 0.187\t Average Accuracy: 0.603\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6v46b-5eHSO",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que, como cabría de esperar, el desempeño mejora sustancialmente en comparación al experimento anterior: hay una ganancia considerable respecto a las métricas de AUC y Kappa, prácticamente sin perjudicar la Accuracy del modelo.\n",
        "\n",
        "Por esta razón, en los experimentos que siguen se considerará esta medida como la estándar para desarrollarlos."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sVz6XEVwGU_S",
        "colab_type": "text"
      },
      "source": [
        "### 5.2. Baseline sin custom features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pywUlEkVGyh_",
        "colab_type": "text"
      },
      "source": [
        "A modo de sanity check, en esta sección se muestra la performance del modelo empleado en el baseline, pero sin los features personalizados. Es decir, solo se considera la vectorización en unigramas para entrenar un clasificador Naive Bayes. La razón de esto es verificar dos cosas:\n",
        "\n",
        "*   Que Naive Bayes funciona y tiene validez aún cuando algunos features empleados no están asociados a términos del Corpus. Es decir, que emplear un conteo del símbolo '#' como un feature de los inputs puede interpretarse en el contexto de Naive Bayes como la probabilidad de que ese feature registre cierta cantidad dada una clase.\n",
        "*   Que estos features personalizados son efectivamente capaces de capturar información relevante del problema y, en consecuencia, mejorar el desempeño del clasificador. En este sentido se esperaría que este modelo tenga un peor desempeño que el baseline dado.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IuFZTb6eI0f_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_1_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kZaLJA-UI5lF",
        "colab_type": "code",
        "outputId": "6af3700f-413e-4342-a3de-6bc73aadedaa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_1_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 189  10]\n",
            " [  0  40  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.79       204\n",
            "        high       0.58      0.26      0.36        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.60      0.43      0.44       311\n",
            "weighted avg       0.64      0.67      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.654\tKappa: 0.177\tAccuracy: 0.672\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  73   4]\n",
            " [ 16 200  15]\n",
            " [  1  67  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.19      0.28        95\n",
            "      medium       0.59      0.87      0.70       231\n",
            "        high       0.53      0.24      0.33        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.54      0.43      0.43       415\n",
            "weighted avg       0.56      0.58      0.52       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.647\tKappa: 0.159\tAccuracy: 0.576\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 16  54   2]\n",
            " [ 11 138  12]\n",
            " [  2  37  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.22      0.32        72\n",
            "      medium       0.60      0.86      0.71       161\n",
            "        high       0.65      0.40      0.50        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.60      0.49      0.51       298\n",
            "weighted avg       0.60      0.60      0.57       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.719\tKappa: 0.256\tAccuracy: 0.604\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  7  59   3]\n",
            " [  4 137   9]\n",
            " [  1  50  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.10      0.17        69\n",
            "      medium       0.56      0.91      0.69       150\n",
            "        high       0.54      0.22      0.31        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.56      0.41      0.39       284\n",
            "weighted avg       0.56      0.56      0.48       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.65\tKappa: 0.132\tAccuracy: 0.556\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.667\t Average Kappa: 0.181\t Average Accuracy: 0.602\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mfy2VvMtN3EZ",
        "colab_type": "text"
      },
      "source": [
        "Al observar los scores promedios obtenidos se aprecia que efectivamente los features considerados en el baseline permiten mejorar el desempeño, aunque sea en forma leve, puesto que si bien las métricas entre uno y otro modelo son difieren en centécimas, para el caso del baseline todas son consistentemente mayores a las observadas en este último modelo."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "II1TUeivJfTU",
        "colab_type": "text"
      },
      "source": [
        "### 5.3. Naive Bayes + Emoji Handler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xgy8bzoRJmxC",
        "colab_type": "text"
      },
      "source": [
        "Tal como se indica en las recomendaciones, se plantea incluir features basados en emojis. Basados en la implenteción de [`twitter-sentiment-analysis`](https://github.com/abdulfatir/twitter-sentiment-analysis/blob/master/code/preprocess.py) y añadiendo el módulo `emoji`.\n",
        "\n",
        "Los features añadidos se basan establecer un listado de emojis y secuencias del estilo de `':)'` según el cual estos símbolos se parsean temporalmente como emociones positivas (`'EMO_POS'`) y negativas (`'EMO_NEG'`), para luego contar la cantidad de apariciones de cada uno de estos tokens. Se utiliza además el contador nativo del módulo `emoji` (`emoji.emoji_count`), lo cual permitiría incorporar parte de la información presente en los emojis no considerados en el listado anteriormente mencionado.\n",
        "\n",
        "En la misma línea del baseline, se considera el modelo Naive Bayes\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2No7L-t2M6xM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class EmojiHandler(BaseEstimator, TransformerMixin):\n",
        "    def parse_emotions(self, tweet):\n",
        "        '''Combina el handler de\n",
        "        https://github.com/abdulfatir/twitter-sentiment-analysis/blob/master/code/preprocess.py\n",
        "        con el módulo emoji'''\n",
        "        # Smile -- :), : ), :-), (:, ( :, (-:, :')\n",
        "        tweet = re.sub(r'(:\\s?\\)|:-\\)|\\(\\s?:|\\(-:|:\\'\\))', ' EMO_POS ', tweet)\n",
        "        tweet = re.sub('😀|😁|😃|😄|😊|🤗|🙂|😋|😌|🤭|🤠', ' EMO_POS ', tweet)\n",
        "        # Laugh -- :D, : D, :-D, xD, x-D, XD, X-D\n",
        "        tweet = re.sub(r'(:\\s?D|:-D|x-?D|X-?D)', ' EMO_POS ', tweet)\n",
        "        tweet = re.sub('😂|🤣|😆',' EMO_POS ', tweet)\n",
        "        # Love -- <3, :*\n",
        "        tweet = re.sub(r'(<3|:\\*)', ' EMO_POS ', tweet)\n",
        "        tweet = re.sub('😍|🥰|❤|🧡|💛|💚|💙|💜|🤎|🖤|🤍|❣|💕|💞|💓|💗|💖|💘|💝|💟',' EMO_POS ', tweet)\n",
        "        # Wink -- ;-), ;), ;-D, ;D, (;,  (-;\n",
        "        tweet = re.sub(r'(;-?\\)|;-?D|\\(-?;)', ' EMO_POS ', tweet)\n",
        "        tweet = re.sub('😘|😙|😚|👍|💯|💪|✌|🤘|🤙|👏|🙌|👊', ' EMO_POS ', tweet)\n",
        "        # Sad -- :-(, : (, :(, ):, )-:\n",
        "        tweet = re.sub(r'(:\\s?\\(|:-\\(|\\)\\s?:|\\)-:)', ' EMO_NEG ', tweet)\n",
        "        tweet = re.sub('😒|😓|😔|😕|☹|🙁|😖|😞|😟|😩',' EMO_NEG ', tweet)\n",
        "        # Cry -- :,(, :'(, :\"(\n",
        "        tweet = re.sub(r'(:,\\(|:\\'\\(|:\"\\()', ' EMO_NEG ', tweet)\n",
        "        tweet = re.sub('😢|😭', ' EMO_NEG ', tweet)\n",
        "        # Other negative emojis\n",
        "        tweet = re.sub('😠|😡|🤬|🤢|🤮|💢|😤|👎|🖕', ' EMO_NEG ', tweet)\n",
        "        return tweet\n",
        "    \n",
        "    def count(self, tweet):\n",
        "        emoji_count = emoji.emoji_count(tweet)\n",
        "        tweet = self.parse_emotions(tweet)\n",
        "        pos_count = tweet.count('EMO_POS')\n",
        "        neg_count = tweet.count('EMO_NEG')\n",
        "        return [pos_count, neg_count, emoji_count]\n",
        "    \n",
        "    def transform(self, X, y=None):\n",
        "        chars = []\n",
        "        for tweet in X:\n",
        "            chars.append(self.count(tweet))\n",
        "        return np.array(chars)\n",
        "    \n",
        "    def fit(self, X, y=None):\n",
        "        return self"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OiuLh1nnM9tJ",
        "colab_type": "code",
        "outputId": "ce499fd4-e16e-4176-d454-0d9f32008def",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Veamos que sucede si ejecutamos el transformer\n",
        "sample = train['anger'].sample(5).tweet\n",
        "pd.DataFrame(zip(sample, EmojiHandler().transform(sample)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Kik to trade, have fun or a conversation  (kik...</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>when you find out the initiative isn't even a ...</td>\n",
              "      <td>[0, 0, 1]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Projection is perception. See it in someone el...</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Need a new outlet for</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>So I wished my sis 12 midnight but received no...</td>\n",
              "      <td>[0, 0, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                   0          1\n",
              "0  Kik to trade, have fun or a conversation  (kik...  [0, 0, 0]\n",
              "1  when you find out the initiative isn't even a ...  [0, 0, 1]\n",
              "2  Projection is perception. See it in someone el...  [0, 0, 0]\n",
              "3                             Need a new outlet for   [0, 0, 0]\n",
              "4  So I wished my sis 12 midnight but received no...  [0, 0, 0]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uypoq8ytNgPI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_2_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EnI7O3RNOb7U",
        "colab_type": "code",
        "outputId": "53921b19-fac9-4882-e61e-5dcbd31dff31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_2_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 190   9]\n",
            " [  1  40  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.11      0.18        53\n",
            "      medium       0.69      0.93      0.79       204\n",
            "        high       0.59      0.24      0.34        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.59      0.43      0.44       311\n",
            "weighted avg       0.64      0.67      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.656\tKappa: 0.174\tAccuracy: 0.672\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 20  72   3]\n",
            " [ 16 200  15]\n",
            " [  1  67  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.54      0.21      0.30        95\n",
            "      medium       0.59      0.87      0.70       231\n",
            "        high       0.54      0.24      0.33        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.56      0.44      0.44       415\n",
            "weighted avg       0.57      0.58      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.652\tKappa: 0.169\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 19  51   2]\n",
            " [ 13 135  13]\n",
            " [  3  33  29]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.54      0.26      0.36        72\n",
            "      medium       0.62      0.84      0.71       161\n",
            "        high       0.66      0.45      0.53        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.61      0.52      0.53       298\n",
            "weighted avg       0.61      0.61      0.59       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.724\tKappa: 0.288\tAccuracy: 0.614\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  9  57   3]\n",
            " [  4 136  10]\n",
            " [  2  49  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.13      0.21        69\n",
            "      medium       0.56      0.91      0.69       150\n",
            "        high       0.52      0.22      0.30        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.56      0.42      0.40       284\n",
            "weighted avg       0.56      0.56      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.658\tKappa: 0.146\tAccuracy: 0.56\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.672\t Average Kappa: 0.194\t Average Accuracy: 0.607\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E4A3I49aOsBo",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que, en promedio, los features diseñados en base a emojis presentados permiten mejorar tanto el modelo visto en la sección 5.2 como el propio baseline. De hecho, supera al baseline en forma incluso mayor en la que este último suepera al modelo que solo considera unigramas sin ningún feature añadido."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gxLDlztkCv_p",
        "colab_type": "text"
      },
      "source": [
        "### 5.4. Naive Bayes + Modified CharsCountTransformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hDE0tAeWDEzj",
        "colab_type": "text"
      },
      "source": [
        "Acá se considera mejorar el feature extractor dado en el baseline por uno que considere también las menciones a otros usuarios (por medio de contar los usos de `'@'`) y del conteo de los puntos suspensivos usados en cada tweet.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H2CrE4aIDinM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ModifiedCharsCountTransformer(BaseEstimator, TransformerMixin):\n",
        "    def get_relevant_chars(self, tweet):\n",
        "        num_hashtags = tweet.count('#')\n",
        "        num_exclamations = tweet.count('!')\n",
        "        num_interrogations = tweet.count('?')\n",
        "        num_mentions = tweet.count('@')\n",
        "        num_dots = tweet.count('...')\n",
        "        return [num_hashtags, num_exclamations, num_interrogations,\n",
        "                num_mentions, num_dots]\n",
        "\n",
        "    def transform(self, X, y=None):\n",
        "        chars = []\n",
        "        for tweet in X:\n",
        "            chars.append(self.get_relevant_chars(tweet))\n",
        "\n",
        "        return np.array(chars)\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "        return self"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OqDkHsxYD4nh",
        "colab_type": "code",
        "outputId": "5d79d72f-f154-485b-a200-a15555ca6b96",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Veamos que sucede si ejecutamos el transformer\n",
        "sample = train['anger'].sample(5).tweet\n",
        "pd.DataFrame(zip(sample, ModifiedCharsCountTransformer().transform(sample)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>@StarFlowerz17 @DreamerAbe #bitter. Nicole and...</td>\n",
              "      <td>[1, 0, 0, 2, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>@ChronAVT ummm, the blog says 'with Simon Steh...</td>\n",
              "      <td>[1, 0, 0, 1, 1]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>You're so thirsty for the chance to disagree w...</td>\n",
              "      <td>[0, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>I get so angry at people that don't know that ...</td>\n",
              "      <td>[1, 0, 0, 0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>@jamiesonhelen @MarianKeyes all three of my ag...</td>\n",
              "      <td>[0, 0, 0, 2, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                   0                1\n",
              "0  @StarFlowerz17 @DreamerAbe #bitter. Nicole and...  [1, 0, 0, 2, 0]\n",
              "1  @ChronAVT ummm, the blog says 'with Simon Steh...  [1, 0, 0, 1, 1]\n",
              "2  You're so thirsty for the chance to disagree w...  [0, 0, 0, 0, 0]\n",
              "3  I get so angry at people that don't know that ...  [1, 0, 0, 0, 0]\n",
              "4  @jamiesonhelen @MarianKeyes all three of my ag...  [0, 0, 0, 2, 0]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0h8h5W_PEDQg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_3_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nupz0SrGFldz",
        "colab_type": "code",
        "outputId": "84cd301b-46d8-4f3d-ddb2-cae93b929bba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_3_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 189  10]\n",
            " [  0  42  12]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.78       204\n",
            "        high       0.55      0.22      0.32        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.59      0.42      0.43       311\n",
            "weighted avg       0.63      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.659\tKappa: 0.154\tAccuracy: 0.666\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 16  76   3]\n",
            " [ 12 206  13]\n",
            " [  1  70  18]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.17      0.26        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.53      0.20      0.29        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.56      0.42      0.42       415\n",
            "weighted avg       0.57      0.58      0.52       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.653\tKappa: 0.147\tAccuracy: 0.578\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  52   2]\n",
            " [ 15 135  11]\n",
            " [  2  37  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.25      0.34        72\n",
            "      medium       0.60      0.84      0.70       161\n",
            "        high       0.67      0.40      0.50        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.50      0.51       298\n",
            "weighted avg       0.60      0.60      0.57       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.727\tKappa: 0.256\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  8  58   3]\n",
            " [  5 137   8]\n",
            " [  3  47  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.12      0.19        69\n",
            "      medium       0.57      0.91      0.70       150\n",
            "        high       0.58      0.23      0.33        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.55      0.42      0.41       284\n",
            "weighted avg       0.55      0.56      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.653\tKappa: 0.153\tAccuracy: 0.563\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.673\t Average Kappa: 0.177\t Average Accuracy: 0.602\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9kLXHVsKGDKJ",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que, en promedio, el desempeño del modelo con esta modificación al extractor `CharsCountTransformer` permite mejorar la métrica AUC, pero en desmedro de las otras dos, particularmente de la Kappa."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ySTx3UzIxiTi",
        "colab_type": "text"
      },
      "source": [
        "### 5.5. Naive Bayes + Bing Liu Count Extractor"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zoQG75BCxpuu",
        "colab_type": "text"
      },
      "source": [
        "Los corpus de palabras positivas y negativas presentados en el enunciado corresponden al `opinion_lexicon` de `nltk`. La idea es utilizarlo registrar como features la cantidad de palabras con connotación positiva y/o negativa en cada tweet.\n",
        "\n",
        "Este extractor de características ya se encuentra implementado en [AffectiveTweets](https://affectivetweets.cms.waikato.ac.nz/benchmark/#logistic-regression-model-using-word-n-grams-bing-lius-lexicon). Notar que, al depender del tokenizador este es un hiperparámetro más a considerar. Por esta razón se realizan pruebas sobre las distintas formas en que el tokenizador `TweetTokenizer` puede operar, más precisamente sobre el manejo de las mayúsculas."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T2PvWsUKT5gh",
        "colab_type": "code",
        "outputId": "d95fe1ef-c05b-4a9e-8f5e-4d913a8d1fea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "#!wget http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar\n",
        "#get_ipython().system_raw(\"unrar x opinion-lexicon-English.rar\")\n",
        "import nltk\n",
        "from nltk.tokenize import TweetTokenizer\n",
        "nltk.download('opinion_lexicon')\n",
        "\n",
        "posLexicon = nltk.corpus.opinion_lexicon.positive()\n",
        "negLexicon = nltk.corpus.opinion_lexicon.negative()\n",
        "num_lines_pos = len(posLexicon)\n",
        "num_lines_neg = len(negLexicon)\n",
        "print(num_lines_pos)\n",
        "print(num_lines_neg)\n",
        "\n",
        "class LiuFeatureExtractor(BaseEstimator, TransformerMixin):\n",
        "    \"\"\"Takes in a corpus of tweets and calculates features using Bing Liu's lexicon\"\"\"\n",
        "    \n",
        "    def __init__(self, tokenizer = TweetTokenizer(preserve_case=True)):\n",
        "      opinion_lexicon = nltk.corpus.opinion_lexicon\n",
        "      self.tokenizer = tokenizer\n",
        "      self.pos_set = set(opinion_lexicon.positive())\n",
        "      self.neg_set = set(opinion_lexicon.negative())\n",
        "\n",
        "    def liu_score(self,sentence):\n",
        "      \"\"\"Calculates the number of positive and negative words in the sentence using Bing Liu's Lexicon\"\"\" \n",
        "      tokenized_sent = self.tokenizer.tokenize(sentence)\n",
        "      pos_words = 0\n",
        "      neg_words = 0\n",
        "      for word in tokenized_sent:\n",
        "        if word in self.pos_set:\n",
        "          pos_words += 1\n",
        "        elif word in self.neg_set:\n",
        "          neg_words += 1\n",
        "      return [pos_words,neg_words]\n",
        "\n",
        "    def transform(self, X, y=None):\n",
        "      \"\"\"Applies liu_score and vader_score on a data.frame containing tweets \"\"\"\n",
        "      values = []\n",
        "      for tweet in X:\n",
        "        values.append(self.liu_score(tweet))\n",
        "\n",
        "      return(np.array(values))\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "      \"\"\"This function must return `self` unless we expect the transform function to perform a \n",
        "      different action on training and testing partitions (e.g., when we calculate unigram features, \n",
        "      the dictionary is only extracted from the first batch)\"\"\"\n",
        "      return self"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package opinion_lexicon to /root/nltk_data...\n",
            "[nltk_data]   Package opinion_lexicon is already up-to-date!\n",
            "2006\n",
            "4783\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3364k37k7r30",
        "colab_type": "code",
        "outputId": "373a091f-57e2-4c8a-9e93-4ed9c8c45e6b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Veamos que sucede si ejecutamos el transformer\n",
        "sample = train['anger'].sample(5).tweet\n",
        "pd.DataFrame(zip(sample, LiuFeatureExtractor().transform(sample)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>the girl sitting in front of me is chewing her...</td>\n",
              "      <td>[2, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Taking a break from the #wedding to #rage at t...</td>\n",
              "      <td>[0, 1]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>I've always had a weird grudge against stingra...</td>\n",
              "      <td>[0, 3]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@FrizzeyesJr meeee, i'm lvl 320 and i have not...</td>\n",
              "      <td>[0, 1]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Why does @dapperlaughs have to come to Glasgow...</td>\n",
              "      <td>[0, 1]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                   0       1\n",
              "0  the girl sitting in front of me is chewing her...  [2, 0]\n",
              "1  Taking a break from the #wedding to #rage at t...  [0, 1]\n",
              "2  I've always had a weird grudge against stingra...  [0, 3]\n",
              "3  @FrizzeyesJr meeee, i'm lvl 320 and i have not...  [0, 1]\n",
              "4  Why does @dapperlaughs have to come to Glasgow...  [0, 1]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q1HTDdKHzZnq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_4_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=True)))\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KVxaxRS7z-7l",
        "colab_type": "code",
        "outputId": "fe7c560f-c279-47e4-f9c9-2dea05da7bde",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_4_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 190  10]\n",
            " [  0  44  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.78       204\n",
            "        high       0.50      0.19      0.27        54\n",
            "\n",
            "    accuracy                           0.66       311\n",
            "   macro avg       0.59      0.41      0.41       311\n",
            "weighted avg       0.63      0.66      0.59       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.647\tKappa: 0.136\tAccuracy: 0.662\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 20  73   2]\n",
            " [ 14 202  15]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.21      0.31        95\n",
            "      medium       0.59      0.87      0.70       231\n",
            "        high       0.54      0.22      0.32        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.57      0.44      0.44       415\n",
            "weighted avg       0.57      0.58      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.663\tKappa: 0.169\tAccuracy: 0.583\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 15  56   1]\n",
            " [ 12 138  11]\n",
            " [  2  40  23]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.21      0.30        72\n",
            "      medium       0.59      0.86      0.70       161\n",
            "        high       0.66      0.35      0.46        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.59      0.47      0.49       298\n",
            "weighted avg       0.59      0.59      0.55       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.721\tKappa: 0.223\tAccuracy: 0.591\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  8  59   2]\n",
            " [  3 139   8]\n",
            " [  0  52  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.73      0.12      0.20        69\n",
            "      medium       0.56      0.93      0.70       150\n",
            "        high       0.57      0.20      0.30        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.62      0.41      0.40       284\n",
            "weighted avg       0.60      0.56      0.48       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.662\tKappa: 0.139\tAccuracy: 0.563\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.673\t Average Kappa: 0.167\t Average Accuracy: 0.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s2aZ0rXe0n4K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_5_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6eSQ4aw70soM",
        "colab_type": "code",
        "outputId": "9a93acd7-9d36-489d-83a4-f1f716e844ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_5_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 190  10]\n",
            " [  0  44  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.78       204\n",
            "        high       0.50      0.19      0.27        54\n",
            "\n",
            "    accuracy                           0.66       311\n",
            "   macro avg       0.59      0.41      0.41       311\n",
            "weighted avg       0.63      0.66      0.59       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.647\tKappa: 0.136\tAccuracy: 0.662\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 20  74   1]\n",
            " [ 14 205  12]\n",
            " [  1  70  18]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.21      0.31        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.58      0.20      0.30        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.58      0.43      0.44       415\n",
            "weighted avg       0.58      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.663\tKappa: 0.165\tAccuracy: 0.586\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 15  56   1]\n",
            " [ 12 139  10]\n",
            " [  2  39  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.21      0.30        72\n",
            "      medium       0.59      0.86      0.70       161\n",
            "        high       0.69      0.37      0.48        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.60      0.48      0.49       298\n",
            "weighted avg       0.60      0.60      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.723\tKappa: 0.235\tAccuracy: 0.597\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  8  59   2]\n",
            " [  3 140   7]\n",
            " [  0  54  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.73      0.12      0.20        69\n",
            "      medium       0.55      0.93      0.69       150\n",
            "        high       0.55      0.17      0.26        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.61      0.41      0.38       284\n",
            "weighted avg       0.59      0.56      0.47       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.663\tKappa: 0.127\tAccuracy: 0.56\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.674\t Average Kappa: 0.166\t Average Accuracy: 0.601\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pxP1pM7n2PfA",
        "colab_type": "text"
      },
      "source": [
        "Se pueden distinguir dos cosas:\n",
        "\n",
        "* Al igual que lo visto en 5.4. este feature extractor permite elevar la performance en AUC respecto al uso de `EmojiHandler`, pero perjudicando importantemente las otras métricas, particularmente la Kappa.\n",
        "\n",
        "* En este caso el `parámetro preserve_case =  False` permite alcanzar una performance ligeramente mayor que al darle el valor `True`, aunque en forma muy leve y en desmedro del desempeño en cuanto a la métrica Kappa."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IVyLO9lp48t9",
        "colab_type": "text"
      },
      "source": [
        "### 5.6. Naive Bayes + Emphasis Handler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jy5qpA7H5Fz7",
        "colab_type": "text"
      },
      "source": [
        "Consideremos como \"Emphasis Handler\" a las formas de escribir una palabra para que esta conlleve un valor semántico extra, como el uso de mayúsculas o duplicar letras. Para esto, se recurre nuevamente a contadores: uno para la cantidad de palabras escritas en mayúsculas y otro para las palabras con letras duplicadas.\n",
        "\n",
        "Dado que es natural encontrar palabras con letras duplicadas en el lenguaje, se considera como palabra con letras duplicadas si alguna letra se repite 3 o más veces consecutivas (sin contar menciones a otros usuarios)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GG_EPqHI6oMe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class EmphasisHandler(BaseEstimator, TransformerMixin):\n",
        "  def count(self, tweet):\n",
        "    tweet = tweet.split(' ')\n",
        "    num_upper = 0\n",
        "    num_long = 0\n",
        "    for i in range(len(tweet)-1):\n",
        "      try:\n",
        "        if (tweet[i])[0] == '@':\n",
        "          tweet.pop(i)\n",
        "        else:\n",
        "          if tweet[i].isupper() and len(tweet[i]) >= 2:\n",
        "            num_upper += 1\n",
        "          elong = re.compile(\"([a-zA-Z])\\\\1{2,}\")\n",
        "          if bool(elong.search(tweet[i])):\n",
        "            num_long += 1\n",
        "      except Exception:\n",
        "        pass\n",
        "    return [num_upper, num_long]\n",
        "    \n",
        "  def transform(self, X, y=None):\n",
        "    chars = []\n",
        "    for tweet in X:\n",
        "      chars.append(self.count(tweet))\n",
        "    \n",
        "    return np.array(chars)\n",
        "\n",
        "  def fit(self, X, y=None):\n",
        "    return self"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AuT2Tdoq8Gs9",
        "colab_type": "code",
        "outputId": "6a4759a5-0ca6-4eca-85b9-97410aee0958",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        }
      },
      "source": [
        "# Veamos que sucede si ejecutamos el transformer\n",
        "sample = train['anger'].tweet\n",
        "pd.DataFrame(zip(sample, EmphasisHandler().transform(sample)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>How the fu*k! Who the heck! moved my fridge!.....</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>So my Indian Uber driver just called someone t...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@DPD_UK I asked for my parcel to be delivered ...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>so ef whichever butt wipe pulled the fire alar...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Don't join @BTCare they put the phone down on ...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>936</th>\n",
              "      <td>@Jen_ny69 People will always get offended ever...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>937</th>\n",
              "      <td>@gayla_weeks1 I try not to let my anger seep i...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>938</th>\n",
              "      <td>I hope my hustle don't offend nobody</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>939</th>\n",
              "      <td>Just watched Django Unchained, Other people ma...</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>940</th>\n",
              "      <td>Lol little things like that make me so angry x</td>\n",
              "      <td>[0, 0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>941 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                     0       1\n",
              "0    How the fu*k! Who the heck! moved my fridge!.....  [0, 0]\n",
              "1    So my Indian Uber driver just called someone t...  [0, 0]\n",
              "2    @DPD_UK I asked for my parcel to be delivered ...  [0, 0]\n",
              "3    so ef whichever butt wipe pulled the fire alar...  [0, 0]\n",
              "4    Don't join @BTCare they put the phone down on ...  [0, 0]\n",
              "..                                                 ...     ...\n",
              "936  @Jen_ny69 People will always get offended ever...  [0, 0]\n",
              "937  @gayla_weeks1 I try not to let my anger seep i...  [0, 0]\n",
              "938               I hope my hustle don't offend nobody  [0, 0]\n",
              "939  Just watched Django Unchained, Other people ma...  [0, 0]\n",
              "940     Lol little things like that make me so angry x  [0, 0]\n",
              "\n",
              "[941 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MTZBj0Kd8kjd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_6_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', EmphasisHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WjLsZJp38v9b",
        "colab_type": "code",
        "outputId": "47abba01-a80a-4756-fb20-3756b944ad76",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_6_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 189  10]\n",
            " [  1  40  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.11      0.18        53\n",
            "      medium       0.68      0.93      0.79       204\n",
            "        high       0.57      0.24      0.34        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.58      0.43      0.44       311\n",
            "weighted avg       0.63      0.67      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.649\tKappa: 0.169\tAccuracy: 0.669\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 19  74   2]\n",
            " [ 16 201  14]\n",
            " [  1  67  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.20      0.29        95\n",
            "      medium       0.59      0.87      0.70       231\n",
            "        high       0.57      0.24      0.33        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.56      0.44      0.44       415\n",
            "weighted avg       0.57      0.58      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.637\tKappa: 0.165\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 16  54   2]\n",
            " [ 12 137  12]\n",
            " [  3  36  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.22      0.31        72\n",
            "      medium       0.60      0.85      0.71       161\n",
            "        high       0.65      0.40      0.50        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.49      0.50       298\n",
            "weighted avg       0.59      0.60      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.719\tKappa: 0.252\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  7  59   3]\n",
            " [  4 137   9]\n",
            " [  2  50  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.54      0.10      0.17        69\n",
            "      medium       0.56      0.91      0.69       150\n",
            "        high       0.52      0.20      0.29        65\n",
            "\n",
            "    accuracy                           0.55       284\n",
            "   macro avg       0.54      0.40      0.38       284\n",
            "weighted avg       0.54      0.55      0.47       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.648\tKappa: 0.125\tAccuracy: 0.553\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.663\t Average Kappa: 0.178\t Average Accuracy: 0.601\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BxjtOGo0D6Dw",
        "colab_type": "text"
      },
      "source": [
        "Vemos que, en términos generales, la implementación de estas características no es más informativa en comparación a las vistas anteriormente. De hecho, el desempeño de esta configuración es peor que el baseline tras incorporar la estratificación.\n",
        "\n",
        "Si se estudia el dataset completo de entrenamiento, uno de los pocos datos donde estas features cobran relevancia son:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YAYr8d93TE4d",
        "colab_type": "code",
        "outputId": "f067c6d5-4164-48e7-f78e-1321bbd103f4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "tweet1 = train['anger'].tweet[16]\n",
        "tweet2 = train['anger'].tweet[63]\n",
        "print(tweet1)\n",
        "print(tweet2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "@ArizonaCoyotes not to mention the GRA guy stops me but let's the 2 ppl in front of me go. WTF. My blood is boiling.\n",
            "OOOOOOOOH MY GOD UUUUGGGGHHHHHHHHH \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "slbVi_wsTOU_",
        "colab_type": "text"
      },
      "source": [
        "La gran mayoría de datos en el dataset donde este extractor entrega un conteo mayor a 0 son aquellos que contienen siglas como ESPN, que intuitivamente son poco informativos respecto a la intensidad del sentimiento.\n",
        "\n",
        "Evidentemente descartar estos features sobre el conjunto de entrenamiento puede ser apresurado, pero considerando que este conjunto debiera ser representativo del fenómeno estudiado es plausible, por más que la intuición humana pueda indicar que debieran ser particularmente útiles al tratarse de oraciones producidas por humanos en internet.\n",
        "\n",
        "Ya teniendo estos features, se procede a continuación a combinarlos, priorizando aquellos que hayan resultado particularmente informativos como el `EmojiHandler`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lqIDBKLGTIR4",
        "colab_type": "text"
      },
      "source": [
        "### 5.7. Baseline Naive Bayes + Emoji Handler"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vIHAxCueVr8l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_7_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oxEF4JjXV-BP",
        "colab_type": "code",
        "outputId": "ebcbc7c0-e9cc-4542-a22f-465fc231b7e0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_7_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 188  11]\n",
            " [  2  38  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.46      0.11      0.18        53\n",
            "      medium       0.69      0.92      0.79       204\n",
            "        high       0.56      0.26      0.35        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.57      0.43      0.44       311\n",
            "weighted avg       0.63      0.67      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.655\tKappa: 0.178\tAccuracy: 0.669\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 19  74   2]\n",
            " [ 14 205  12]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.20      0.29        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.59      0.22      0.33        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.58      0.44      0.44       415\n",
            "weighted avg       0.58      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.651\tKappa: 0.173\tAccuracy: 0.588\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  51   3]\n",
            " [ 15 135  11]\n",
            " [  2  35  28]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.25      0.34        72\n",
            "      medium       0.61      0.84      0.71       161\n",
            "        high       0.67      0.43      0.52        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.60      0.51      0.52       298\n",
            "weighted avg       0.60      0.61      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.727\tKappa: 0.273\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  56   3]\n",
            " [  5 133  12]\n",
            " [  2  48  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.14      0.23        69\n",
            "      medium       0.56      0.89      0.69       150\n",
            "        high       0.50      0.23      0.32        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.55      0.42      0.41       284\n",
            "weighted avg       0.55      0.56      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.662\tKappa: 0.148\tAccuracy: 0.556\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.674\t Average Kappa: 0.193\t Average Accuracy: 0.605\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3FN-eUdKCqwv",
        "colab_type": "text"
      },
      "source": [
        "Veamos qué pasa si escogemos la versión modificada de `CharsCountTransformer`, `ModifiedCharsCountTransformer`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w4fZqXsJYr59",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_8_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LJB63fdnYsB_",
        "colab_type": "code",
        "outputId": "6cb132b6-0d07-46e6-de06-d412b82932be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_8_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 190   9]\n",
            " [  1  42  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.11      0.18        53\n",
            "      medium       0.68      0.93      0.79       204\n",
            "        high       0.55      0.20      0.30        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.58      0.42      0.42       311\n",
            "weighted avg       0.63      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.662\tKappa: 0.151\tAccuracy: 0.666\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  75   2]\n",
            " [ 12 206  13]\n",
            " [  1  69  19]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.19      0.29        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.56      0.21      0.31        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.58      0.43      0.43       415\n",
            "weighted avg       0.58      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.659\tKappa: 0.164\tAccuracy: 0.586\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  52   2]\n",
            " [ 14 136  11]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.25      0.34        72\n",
            "      medium       0.61      0.84      0.71       161\n",
            "        high       0.68      0.42      0.51        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.60      0.50      0.52       298\n",
            "weighted avg       0.60      0.61      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.73\tKappa: 0.269\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  9  57   3]\n",
            " [  5 135  10]\n",
            " [  3  47  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.13      0.21        69\n",
            "      medium       0.56      0.90      0.69       150\n",
            "        high       0.54      0.23      0.32        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.54      0.42      0.41       284\n",
            "weighted avg       0.55      0.56      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.66\tKappa: 0.151\tAccuracy: 0.56\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.678\t Average Kappa: 0.184\t Average Accuracy: 0.605\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oTgaFz2GCq6o",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que la ganancia en 0.004 en la métrica AUC repercute en una pérdida de 0.009 puntos en Kappa."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QgM5WbZZCrCw",
        "colab_type": "text"
      },
      "source": [
        "### 5.8. Baseline + Stratify + Emoji Handler + Bing Liu Sentiment Lexicon"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BIxHkro2h9Lc",
        "colab_type": "text"
      },
      "source": [
        "Veremos a continuación el efecto de complementar `EmojiHandler` y ambas versiones de `CharCountTransformer` con el extractor de features basado en el lexicon de Bing Liu. Se considera también la opción de variar el parámetro `preserve_case` dada la poca claridad respecto a su efecto, pese a que pareciera que asignarle el valor `False` es más benéfico."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iUWJAeCfhFUf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_9_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=True)))\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9k2xyrzZh6_z",
        "colab_type": "code",
        "outputId": "b06aca03-2112-4a65-b927-d60957cbbe01",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_9_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 190  10]\n",
            " [  0  42  12]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.79       204\n",
            "        high       0.55      0.22      0.32        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.61      0.42      0.43       311\n",
            "weighted avg       0.64      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.649\tKappa: 0.159\tAccuracy: 0.669\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 19  75   1]\n",
            " [ 13 205  13]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.20      0.30        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.59      0.22      0.33        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.58      0.44      0.44       415\n",
            "weighted avg       0.59      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.172\tAccuracy: 0.588\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 13  57   2]\n",
            " [ 16 133  12]\n",
            " [  2  34  29]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.42      0.18      0.25        72\n",
            "      medium       0.59      0.83      0.69       161\n",
            "        high       0.67      0.45      0.54        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.56      0.48      0.49       298\n",
            "weighted avg       0.57      0.59      0.55       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.728\tKappa: 0.232\tAccuracy: 0.587\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  57   2]\n",
            " [  3 138   9]\n",
            " [  1  51  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.71      0.14      0.24        69\n",
            "      medium       0.56      0.92      0.70       150\n",
            "        high       0.54      0.20      0.29        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.61      0.42      0.41       284\n",
            "weighted avg       0.59      0.57      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.671\tKappa: 0.153\tAccuracy: 0.567\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.679\t Average Kappa: 0.179\t Average Accuracy: 0.603\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HMn4QfpVhvy8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_10_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GKKJywAwil_h",
        "colab_type": "code",
        "outputId": "994fba7e-8171-4afc-b767-af9acab1d57b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_10_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 189  11]\n",
            " [  0  43  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.78       204\n",
            "        high       0.50      0.20      0.29        54\n",
            "\n",
            "    accuracy                           0.66       311\n",
            "   macro avg       0.59      0.41      0.42       311\n",
            "weighted avg       0.63      0.66      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.65\tKappa: 0.143\tAccuracy: 0.662\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 19  75   1]\n",
            " [ 13 206  12]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.20      0.30        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.61      0.22      0.33        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.59      0.44      0.45       415\n",
            "weighted avg       0.59      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.175\tAccuracy: 0.59\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 14  57   1]\n",
            " [ 16 134  11]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.44      0.19      0.27        72\n",
            "      medium       0.59      0.83      0.69       161\n",
            "        high       0.69      0.42      0.52        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.57      0.48      0.49       298\n",
            "weighted avg       0.58      0.59      0.55       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.729\tKappa: 0.227\tAccuracy: 0.587\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  57   2]\n",
            " [  3 139   8]\n",
            " [  1  50  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.71      0.14      0.24        69\n",
            "      medium       0.57      0.93      0.70       150\n",
            "        high       0.58      0.22      0.31        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.62      0.43      0.42       284\n",
            "weighted avg       0.61      0.57      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.672\tKappa: 0.167\tAccuracy: 0.574\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.68\t Average Kappa: 0.178\t Average Accuracy: 0.603\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z9DvPIwShy3j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_11_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=True)))\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gSE9-vN7ioeR",
        "colab_type": "code",
        "outputId": "22048062-037d-4911-bdf8-34e9e0db549f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_11_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 191   9]\n",
            " [  0  44  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.94      0.79       204\n",
            "        high       0.53      0.19      0.27        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.60      0.41      0.42       311\n",
            "weighted avg       0.64      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.652\tKappa: 0.141\tAccuracy: 0.666\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  76   1]\n",
            " [ 14 204  13]\n",
            " [  2  70  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.19      0.28        95\n",
            "      medium       0.58      0.88      0.70       231\n",
            "        high       0.55      0.19      0.28        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.55      0.42      0.42       415\n",
            "weighted avg       0.56      0.58      0.52       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.675\tKappa: 0.145\tAccuracy: 0.576\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 14  56   2]\n",
            " [ 14 137  10]\n",
            " [  2  35  28]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.47      0.19      0.27        72\n",
            "      medium       0.60      0.85      0.70       161\n",
            "        high       0.70      0.43      0.53        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.49      0.50       298\n",
            "weighted avg       0.59      0.60      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.734\tKappa: 0.251\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  9  59   1]\n",
            " [  3 140   7]\n",
            " [  1  50  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.69      0.13      0.22        69\n",
            "      medium       0.56      0.93      0.70       150\n",
            "        high       0.64      0.22      0.32        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.63      0.43      0.41       284\n",
            "weighted avg       0.61      0.57      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.672\tKappa: 0.161\tAccuracy: 0.574\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.683\t Average Kappa: 0.174\t Average Accuracy: 0.604\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v0TnZumnh1l3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_12_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DXmpbCA6iqLr",
        "colab_type": "code",
        "outputId": "cd079c61-df80-4b54-c240-c433f06a3d28",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_12_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 191   9]\n",
            " [  0  44  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.94      0.79       204\n",
            "        high       0.53      0.19      0.27        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.60      0.41      0.42       311\n",
            "weighted avg       0.64      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.652\tKappa: 0.141\tAccuracy: 0.666\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 19  75   1]\n",
            " [ 14 205  12]\n",
            " [  2  70  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.54      0.20      0.29        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.57      0.19      0.29        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.57      0.43      0.43       415\n",
            "weighted avg       0.57      0.58      0.52       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.675\tKappa: 0.154\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 15  56   1]\n",
            " [ 14 137  10]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.48      0.21      0.29        72\n",
            "      medium       0.60      0.85      0.70       161\n",
            "        high       0.71      0.42      0.52        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.60      0.49      0.51       298\n",
            "weighted avg       0.60      0.60      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.735\tKappa: 0.249\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  9  59   1]\n",
            " [  3 139   8]\n",
            " [  1  51  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.69      0.13      0.22        69\n",
            "      medium       0.56      0.93      0.70       150\n",
            "        high       0.59      0.20      0.30        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.61      0.42      0.41       284\n",
            "weighted avg       0.60      0.57      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.671\tKappa: 0.148\tAccuracy: 0.567\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.683\t Average Kappa: 0.173\t Average Accuracy: 0.604\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6fPGUacCrLv",
        "colab_type": "text"
      },
      "source": [
        "En vista de los resultados obtenidos hasta ahora, se considera que el top 3 en cuanto a balance entre el promedio de las 3 métricas son el caso estudiado en 5.7 y los que incorporan a las features anteriores los contadores basados en el lexicon de Bing Liu con parámetro `preserve_case = False`.\n",
        "\n",
        "Para este top 3 de modelos estudiaremos a continuación la adición de `EmphasisHandler`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bnhjL5fMhlUi",
        "colab_type": "text"
      },
      "source": [
        "### 5.9. Top 3 + Emphasis Handler"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l8MTK46MpCmx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_13_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "df6fPuZfpll-",
        "colab_type": "code",
        "outputId": "a0898987-4fe7-4f3c-ccdd-b03ad9827b88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_13_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  5 188  11]\n",
            " [  2  38  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.46      0.11      0.18        53\n",
            "      medium       0.69      0.92      0.79       204\n",
            "        high       0.56      0.26      0.35        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.57      0.43      0.44       311\n",
            "weighted avg       0.63      0.67      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.652\tKappa: 0.178\tAccuracy: 0.669\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  75   2]\n",
            " [ 14 205  12]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.19      0.28        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.59      0.22      0.33        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.57      0.43      0.44       415\n",
            "weighted avg       0.58      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.643\tKappa: 0.167\tAccuracy: 0.586\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  51   3]\n",
            " [ 14 136  11]\n",
            " [  2  35  28]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.25      0.34        72\n",
            "      medium       0.61      0.84      0.71       161\n",
            "        high       0.67      0.43      0.52        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.60      0.51      0.52       298\n",
            "weighted avg       0.60      0.61      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.727\tKappa: 0.278\tAccuracy: 0.611\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  56   3]\n",
            " [  5 133  12]\n",
            " [  2  48  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.14      0.23        69\n",
            "      medium       0.56      0.89      0.69       150\n",
            "        high       0.50      0.23      0.32        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.55      0.42      0.41       284\n",
            "weighted avg       0.55      0.56      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.66\tKappa: 0.148\tAccuracy: 0.556\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.67\t Average Kappa: 0.193\t Average Accuracy: 0.605\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xZ1mn8v8pYIx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_14_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Ra1OwPMrmdc",
        "colab_type": "code",
        "outputId": "60e5bf7c-24ff-4930-d0d8-cdeb6944cf3a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_14_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 189  11]\n",
            " [  0  42  12]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.93      0.78       204\n",
            "        high       0.52      0.22      0.31        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.60      0.42      0.43       311\n",
            "weighted avg       0.64      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.647\tKappa: 0.154\tAccuracy: 0.666\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  76   1]\n",
            " [ 14 205  12]\n",
            " [  1  68  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.19      0.28        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.61      0.22      0.33        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.58      0.43      0.44       415\n",
            "weighted avg       0.58      0.59      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.659\tKappa: 0.165\tAccuracy: 0.586\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 14  57   1]\n",
            " [ 16 134  11]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.44      0.19      0.27        72\n",
            "      medium       0.59      0.83      0.69       161\n",
            "        high       0.69      0.42      0.52        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.57      0.48      0.49       298\n",
            "weighted avg       0.58      0.59      0.55       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.729\tKappa: 0.227\tAccuracy: 0.587\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  57   2]\n",
            " [  3 139   8]\n",
            " [  1  51  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.71      0.14      0.24        69\n",
            "      medium       0.56      0.93      0.70       150\n",
            "        high       0.57      0.20      0.30        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.61      0.42      0.41       284\n",
            "weighted avg       0.60      0.57      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.67\tKappa: 0.158\tAccuracy: 0.57\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.676\t Average Kappa: 0.176\t Average Accuracy: 0.602\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "abOQ8YD3pfQ3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_15_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', MultinomialNB())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KXPMrw1usZ1a",
        "colab_type": "code",
        "outputId": "55b8b50b-4236-44ec-dcf2-9be8af0bf0cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_15_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [  4 191   9]\n",
            " [  0  43  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.11      0.19        53\n",
            "      medium       0.68      0.94      0.79       204\n",
            "        high       0.55      0.20      0.30        54\n",
            "\n",
            "    accuracy                           0.67       311\n",
            "   macro avg       0.61      0.42      0.43       311\n",
            "weighted avg       0.64      0.67      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.65\tKappa: 0.152\tAccuracy: 0.669\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 18  76   1]\n",
            " [ 14 205  12]\n",
            " [  2  69  18]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.19      0.28        95\n",
            "      medium       0.59      0.89      0.71       231\n",
            "        high       0.58      0.20      0.30        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.57      0.43      0.43       415\n",
            "weighted avg       0.57      0.58      0.52       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.665\tKappa: 0.154\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 15  56   1]\n",
            " [ 14 137  10]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.48      0.21      0.29        72\n",
            "      medium       0.60      0.85      0.70       161\n",
            "        high       0.71      0.42      0.52        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.60      0.49      0.51       298\n",
            "weighted avg       0.60      0.60      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.734\tKappa: 0.249\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  9  59   1]\n",
            " [  3 139   8]\n",
            " [  1  50  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.69      0.13      0.22        69\n",
            "      medium       0.56      0.93      0.70       150\n",
            "        high       0.61      0.22      0.32        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.62      0.42      0.41       284\n",
            "weighted avg       0.60      0.57      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.67\tKappa: 0.156\tAccuracy: 0.57\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.68\t Average Kappa: 0.178\t Average Accuracy: 0.605\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UVNTr3xshlRv",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que en estas últimas tres pruebas el desempeño al incorporar este último feature extractor decae. No obstante,la última prueba presentada exhibe una performance comparable al top 3, por lo que se incluirá en las pruebas para seleccionar el clasificador final, formando un top 4."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QBeueReshmk7",
        "colab_type": "text"
      },
      "source": [
        "### 5.10. Pruebas en Random Forest "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBIEGdvpi2C4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_16_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P9En76jGjqnG",
        "colab_type": "code",
        "outputId": "83810f9f-9aa7-44e3-8b39-faa420358b05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_16_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  5  48   0]\n",
            " [  9 188   7]\n",
            " [  0  45   9]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.36      0.09      0.15        53\n",
            "      medium       0.67      0.92      0.78       204\n",
            "        high       0.56      0.17      0.26        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.53      0.39      0.39       311\n",
            "weighted avg       0.60      0.65      0.58       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.621\tKappa: 0.103\tAccuracy: 0.65\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 24  71   0]\n",
            " [ 13 213   5]\n",
            " [  0  68  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.25      0.36        95\n",
            "      medium       0.61      0.92      0.73       231\n",
            "        high       0.81      0.24      0.37        89\n",
            "\n",
            "    accuracy                           0.62       415\n",
            "   macro avg       0.69      0.47      0.49       415\n",
            "weighted avg       0.66      0.62      0.57       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.677\tKappa: 0.234\tAccuracy: 0.622\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 16  53   3]\n",
            " [ 10 141  10]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.22      0.32        72\n",
            "      medium       0.61      0.88      0.72       161\n",
            "        high       0.68      0.42      0.51        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.62      0.50      0.52       298\n",
            "weighted avg       0.62      0.62      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.747\tKappa: 0.28\tAccuracy: 0.617\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  7  60   2]\n",
            " [  2 133  15]\n",
            " [  0  43  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.78      0.10      0.18        69\n",
            "      medium       0.56      0.89      0.69       150\n",
            "        high       0.56      0.34      0.42        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.64      0.44      0.43       284\n",
            "weighted avg       0.62      0.57      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.655\tKappa: 0.177\tAccuracy: 0.57\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.675\t Average Kappa: 0.199\t Average Accuracy: 0.615\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pai4X90-jAji",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_17_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dPUObKj6lfR0",
        "colab_type": "code",
        "outputId": "df1828ca-9029-4dd5-d498-8baa02ab5b08",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_17_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  8 190   6]\n",
            " [  0  45   9]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.27      0.06      0.09        53\n",
            "      medium       0.67      0.93      0.78       204\n",
            "        high       0.60      0.17      0.26        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.51      0.38      0.38       311\n",
            "weighted avg       0.59      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.643\tKappa: 0.088\tAccuracy: 0.65\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 30  64   1]\n",
            " [ 15 207   9]\n",
            " [  1  67  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.32      0.43        95\n",
            "      medium       0.61      0.90      0.73       231\n",
            "        high       0.68      0.24      0.35        89\n",
            "\n",
            "    accuracy                           0.62       415\n",
            "   macro avg       0.65      0.48      0.50       415\n",
            "weighted avg       0.64      0.62      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.714\tKappa: 0.251\tAccuracy: 0.622\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  52   2]\n",
            " [ 15 138   8]\n",
            " [  2  38  25]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.25      0.34        72\n",
            "      medium       0.61      0.86      0.71       161\n",
            "        high       0.71      0.38      0.50        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.61      0.50      0.52       298\n",
            "weighted avg       0.61      0.61      0.57       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.73\tKappa: 0.263\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 11  56   2]\n",
            " [  4 133  13]\n",
            " [  0  41  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.73      0.16      0.26        69\n",
            "      medium       0.58      0.89      0.70       150\n",
            "        high       0.62      0.37      0.46        65\n",
            "\n",
            "    accuracy                           0.59       284\n",
            "   macro avg       0.64      0.47      0.47       284\n",
            "weighted avg       0.62      0.59      0.54       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.687\tKappa: 0.226\tAccuracy: 0.592\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.694\t Average Kappa: 0.207\t Average Accuracy: 0.618\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uPLz0F_mjH2-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_18_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_KVhuNKZl4HP",
        "colab_type": "code",
        "outputId": "d785d425-91e3-4d95-f24a-98d606fd7d48",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_18_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  6  47   0]\n",
            " [ 10 187   7]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.38      0.11      0.17        53\n",
            "      medium       0.67      0.92      0.77       204\n",
            "        high       0.53      0.15      0.23        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.53      0.39      0.39       311\n",
            "weighted avg       0.59      0.65      0.58       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.64\tKappa: 0.098\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 31  63   1]\n",
            " [ 17 208   6]\n",
            " [  0  73  16]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.33      0.43        95\n",
            "      medium       0.60      0.90      0.72       231\n",
            "        high       0.70      0.18      0.29        89\n",
            "\n",
            "    accuracy                           0.61       415\n",
            "   macro avg       0.65      0.47      0.48       415\n",
            "weighted avg       0.63      0.61      0.56       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.704\tKappa: 0.229\tAccuracy: 0.614\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 23  45   4]\n",
            " [ 22 132   7]\n",
            " [  4  36  25]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.47      0.32      0.38        72\n",
            "      medium       0.62      0.82      0.71       161\n",
            "        high       0.69      0.38      0.50        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.51      0.53       298\n",
            "weighted avg       0.60      0.60      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.738\tKappa: 0.277\tAccuracy: 0.604\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  57   2]\n",
            " [  6 130  14]\n",
            " [  1  40  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.14      0.23        69\n",
            "      medium       0.57      0.87      0.69       150\n",
            "        high       0.60      0.37      0.46        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.59      0.46      0.46       284\n",
            "weighted avg       0.58      0.58      0.53       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.674\tKappa: 0.204\tAccuracy: 0.577\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.689\t Average Kappa: 0.202\t Average Accuracy: 0.61\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5uYOz72BjLlY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_19_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t8jyXTUjmTCx",
        "colab_type": "code",
        "outputId": "e099df76-335e-40ef-8d4b-7f31169a0d31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_19_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  8 188   8]\n",
            " [  0  45   9]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.08      0.12        53\n",
            "      medium       0.67      0.92      0.77       204\n",
            "        high       0.53      0.17      0.25        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.51      0.39      0.38       311\n",
            "weighted avg       0.59      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.628\tKappa: 0.091\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 30  64   1]\n",
            " [ 15 207   9]\n",
            " [  0  66  23]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.67      0.32      0.43        95\n",
            "      medium       0.61      0.90      0.73       231\n",
            "        high       0.70      0.26      0.38        89\n",
            "\n",
            "    accuracy                           0.63       415\n",
            "   macro avg       0.66      0.49      0.51       415\n",
            "weighted avg       0.64      0.63      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.707\tKappa: 0.262\tAccuracy: 0.627\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  52   2]\n",
            " [ 12 142   7]\n",
            " [  2  39  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.25      0.35        72\n",
            "      medium       0.61      0.88      0.72       161\n",
            "        high       0.73      0.37      0.49        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.63      0.50      0.52       298\n",
            "weighted avg       0.62      0.62      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.741\tKappa: 0.275\tAccuracy: 0.617\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 11  57   1]\n",
            " [  4 132  14]\n",
            " [  1  43  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.69      0.16      0.26        69\n",
            "      medium       0.57      0.88      0.69       150\n",
            "        high       0.58      0.32      0.42        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.61      0.45      0.46       284\n",
            "weighted avg       0.60      0.58      0.52       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.667\tKappa: 0.196\tAccuracy: 0.577\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.686\t Average Kappa: 0.206\t Average Accuracy: 0.617\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "crs2kupbhlPZ",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que al menos en este problema el clasificador Random Forest permite explotar en mejor medida los features encontrados en comparación con Naive Bayes. \n",
        "\n",
        "Más aún, Random Forest permite incorporar n-gramas sin violar los supuestos del clasificador. Por esta razón se procede ahora a probar los clasificadores con bigramas para estudiar el impacto de complejizar el modelo de lenguaje subyacente."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LpvUO7uJhlLG",
        "colab_type": "text"
      },
      "source": [
        "#### 5.10.1 Pruebas con Unigramas y Bigramas"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E4hVKL-TqVrQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_20_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(1, 2))),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rWN0KZgUrT-x",
        "colab_type": "code",
        "outputId": "12e8b959-bbcc-4da3-cc02-78658000900b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_20_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  7 191   6]\n",
            " [  0  47   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.30      0.06      0.10        53\n",
            "      medium       0.66      0.94      0.78       204\n",
            "        high       0.54      0.13      0.21        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.50      0.37      0.36       311\n",
            "weighted avg       0.58      0.65      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.612\tKappa: 0.069\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 24  70   1]\n",
            " [ 10 218   3]\n",
            " [  1  78  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.69      0.25      0.37        95\n",
            "      medium       0.60      0.94      0.73       231\n",
            "        high       0.71      0.11      0.19        89\n",
            "\n",
            "    accuracy                           0.61       415\n",
            "   macro avg       0.67      0.44      0.43       415\n",
            "weighted avg       0.64      0.61      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.696\tKappa: 0.186\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[  9  62   1]\n",
            " [  8 147   6]\n",
            " [  2  42  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.47      0.12      0.20        72\n",
            "      medium       0.59      0.91      0.71       161\n",
            "        high       0.75      0.32      0.45        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.60      0.45      0.45       298\n",
            "weighted avg       0.59      0.59      0.53       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.726\tKappa: 0.202\tAccuracy: 0.594\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  5  63   1]\n",
            " [  3 141   6]\n",
            " [  0  48  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.07      0.13        69\n",
            "      medium       0.56      0.94      0.70       150\n",
            "        high       0.71      0.26      0.38        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.63      0.42      0.40       284\n",
            "weighted avg       0.61      0.57      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.648\tKappa: 0.157\tAccuracy: 0.574\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.67\t Average Kappa: 0.153\t Average Accuracy: 0.605\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8n9FtLO2qcpb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_21_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(2, 2))),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m90rx_rDsARV",
        "colab_type": "code",
        "outputId": "b488405a-0bd0-48a8-ff73-f061aa7f23b9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_21_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  8 190   6]\n",
            " [  0  47   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.27      0.06      0.09        53\n",
            "      medium       0.66      0.93      0.77       204\n",
            "        high       0.54      0.13      0.21        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.49      0.37      0.36       311\n",
            "weighted avg       0.57      0.64      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.525\tKappa: 0.064\tAccuracy: 0.643\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 15  79   1]\n",
            " [ 11 213   7]\n",
            " [  0  78  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.16      0.25        95\n",
            "      medium       0.58      0.92      0.71       231\n",
            "        high       0.58      0.12      0.20        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.58      0.40      0.39       415\n",
            "weighted avg       0.58      0.58      0.49       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.62\tKappa: 0.116\tAccuracy: 0.576\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[38 33  1]\n",
            " [56 97  8]\n",
            " [14 34 17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.35      0.53      0.42        72\n",
            "      medium       0.59      0.60      0.60       161\n",
            "        high       0.65      0.26      0.37        65\n",
            "\n",
            "    accuracy                           0.51       298\n",
            "   macro avg       0.53      0.46      0.46       298\n",
            "weighted avg       0.55      0.51      0.51       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.682\tKappa: 0.178\tAccuracy: 0.51\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  4  64   1]\n",
            " [  3 140   7]\n",
            " [  0  58   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.06      0.11        69\n",
            "      medium       0.53      0.93      0.68       150\n",
            "        high       0.47      0.11      0.18        65\n",
            "\n",
            "    accuracy                           0.53       284\n",
            "   macro avg       0.52      0.37      0.32       284\n",
            "weighted avg       0.53      0.53      0.42       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.54\tKappa: 0.053\tAccuracy: 0.532\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.592\t Average Kappa: 0.103\t Average Accuracy: 0.565\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O1Uyfb3Sqm94",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_22_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(1, 2))),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "neK33BtytRh3",
        "colab_type": "code",
        "outputId": "f736e760-6fdd-454b-822d-83998093d038",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_22_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  8 189   7]\n",
            " [  0  45   9]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.27      0.06      0.09        53\n",
            "      medium       0.67      0.93      0.77       204\n",
            "        high       0.56      0.17      0.26        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.50      0.38      0.38       311\n",
            "weighted avg       0.58      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.599\tKappa: 0.084\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 29  65   1]\n",
            " [ 15 211   5]\n",
            " [  3  76  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.31      0.41        95\n",
            "      medium       0.60      0.91      0.72       231\n",
            "        high       0.62      0.11      0.19        89\n",
            "\n",
            "    accuracy                           0.60       415\n",
            "   macro avg       0.61      0.44      0.44       415\n",
            "weighted avg       0.61      0.60      0.54       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.701\tKappa: 0.195\tAccuracy: 0.602\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 20  51   1]\n",
            " [ 21 135   5]\n",
            " [  5  42  18]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.43      0.28      0.34        72\n",
            "      medium       0.59      0.84      0.69       161\n",
            "        high       0.75      0.28      0.40        65\n",
            "\n",
            "    accuracy                           0.58       298\n",
            "   macro avg       0.59      0.46      0.48       298\n",
            "weighted avg       0.59      0.58      0.55       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.714\tKappa: 0.211\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  3  65   1]\n",
            " [  3 142   5]\n",
            " [  0  49  16]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.04      0.08        69\n",
            "      medium       0.55      0.95      0.70       150\n",
            "        high       0.73      0.25      0.37        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.59      0.41      0.38       284\n",
            "weighted avg       0.58      0.57      0.47       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.136\tAccuracy: 0.567\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.67\t Average Kappa: 0.157\t Average Accuracy: 0.599\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Df7lQG7fqnHO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_23_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(2, 2))),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RoK5lv65uEdp",
        "colab_type": "code",
        "outputId": "95a6eb89-7528-46f8-a34d-8450b31890a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_23_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  9 187   8]\n",
            " [  0  47   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.25      0.06      0.09        53\n",
            "      medium       0.66      0.92      0.77       204\n",
            "        high       0.47      0.13      0.20        54\n",
            "\n",
            "    accuracy                           0.63       311\n",
            "   macro avg       0.46      0.37      0.35       311\n",
            "weighted avg       0.56      0.63      0.55       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.549\tKappa: 0.05\tAccuracy: 0.633\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 19  75   1]\n",
            " [ 14 210   7]\n",
            " [  0  78  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.20      0.30        95\n",
            "      medium       0.58      0.91      0.71       231\n",
            "        high       0.58      0.12      0.20        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.58      0.41      0.40       415\n",
            "weighted avg       0.58      0.58      0.51       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.66\tKappa: 0.131\tAccuracy: 0.578\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 27  44   1]\n",
            " [ 33 121   7]\n",
            " [  4  46  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.42      0.38      0.40        72\n",
            "      medium       0.57      0.75      0.65       161\n",
            "        high       0.65      0.23      0.34        65\n",
            "\n",
            "    accuracy                           0.55       298\n",
            "   macro avg       0.55      0.45      0.46       298\n",
            "weighted avg       0.55      0.55      0.52       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.691\tKappa: 0.174\tAccuracy: 0.547\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  3  65   1]\n",
            " [  4 141   5]\n",
            " [  0  58   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.43      0.04      0.08        69\n",
            "      medium       0.53      0.94      0.68       150\n",
            "        high       0.54      0.11      0.18        65\n",
            "\n",
            "    accuracy                           0.53       284\n",
            "   macro avg       0.50      0.36      0.31       284\n",
            "weighted avg       0.51      0.53      0.42       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.578\tKappa: 0.049\tAccuracy: 0.532\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.619\t Average Kappa: 0.101\t Average Accuracy: 0.573\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RVbVl843q_79",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_24_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(1, 2))),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bZlwvuCIugWZ",
        "colab_type": "code",
        "outputId": "70570e99-8c1b-40d8-8a4f-423aad356a35",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_24_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  8 191   5]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.08      0.12        53\n",
            "      medium       0.67      0.94      0.78       204\n",
            "        high       0.62      0.15      0.24        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.54      0.39      0.38       311\n",
            "weighted avg       0.60      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.641\tKappa: 0.093\tAccuracy: 0.653\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 26  68   1]\n",
            " [  8 216   7]\n",
            " [  1  75  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.74      0.27      0.40        95\n",
            "      medium       0.60      0.94      0.73       231\n",
            "        high       0.62      0.15      0.24        89\n",
            "\n",
            "    accuracy                           0.61       415\n",
            "   macro avg       0.65      0.45      0.46       415\n",
            "weighted avg       0.64      0.61      0.55       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.696\tKappa: 0.21\tAccuracy: 0.614\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 15  56   1]\n",
            " [ 16 141   4]\n",
            " [  3  45  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.44      0.21      0.28        72\n",
            "      medium       0.58      0.88      0.70       161\n",
            "        high       0.77      0.26      0.39        65\n",
            "\n",
            "    accuracy                           0.58       298\n",
            "   macro avg       0.60      0.45      0.46       298\n",
            "weighted avg       0.59      0.58      0.53       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.722\tKappa: 0.19\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  8  61   0]\n",
            " [  4 144   2]\n",
            " [  0  52  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.67      0.12      0.20        69\n",
            "      medium       0.56      0.96      0.71       150\n",
            "        high       0.87      0.20      0.33        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.70      0.43      0.41       284\n",
            "weighted avg       0.66      0.58      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.666\tKappa: 0.161\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.681\t Average Kappa: 0.164\t Average Accuracy: 0.607\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P-R9ipGXq_5b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_25_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(2, 2))),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LkUz9Weru6qY",
        "colab_type": "code",
        "outputId": "fb7dae89-1f07-481d-baba-d76af8d9948a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_25_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  7 190   7]\n",
            " [  0  47   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.30      0.06      0.10        53\n",
            "      medium       0.66      0.93      0.77       204\n",
            "        high       0.50      0.13      0.21        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.49      0.37      0.36       311\n",
            "weighted avg       0.57      0.64      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.582\tKappa: 0.064\tAccuracy: 0.643\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 17  77   1]\n",
            " [ 12 211   8]\n",
            " [  1  76  12]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.18      0.27        95\n",
            "      medium       0.58      0.91      0.71       231\n",
            "        high       0.57      0.13      0.22        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.57      0.41      0.40       415\n",
            "weighted avg       0.57      0.58      0.50       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.664\tKappa: 0.129\tAccuracy: 0.578\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 21  48   3]\n",
            " [ 25 130   6]\n",
            " [  3  45  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.43      0.29      0.35        72\n",
            "      medium       0.58      0.81      0.68       161\n",
            "        high       0.65      0.26      0.37        65\n",
            "\n",
            "    accuracy                           0.56       298\n",
            "   macro avg       0.56      0.45      0.47       298\n",
            "weighted avg       0.56      0.56      0.53       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.7\tKappa: 0.188\tAccuracy: 0.564\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  5  63   1]\n",
            " [  3 140   7]\n",
            " [  0  58   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.07      0.13        69\n",
            "      medium       0.54      0.93      0.68       150\n",
            "        high       0.47      0.11      0.18        65\n",
            "\n",
            "    accuracy                           0.54       284\n",
            "   macro avg       0.54      0.37      0.33       284\n",
            "weighted avg       0.54      0.54      0.43       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.583\tKappa: 0.062\tAccuracy: 0.535\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.632\t Average Kappa: 0.111\t Average Accuracy: 0.58\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mMdJGPDdrKie",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_26_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(1, 2))),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qZcDtBJ4vPDg",
        "colab_type": "code",
        "outputId": "76f2f506-1e33-448f-8fa2-d2db1d0c1dba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_26_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  8 189   7]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.08      0.12        53\n",
            "      medium       0.67      0.93      0.77       204\n",
            "        high       0.53      0.15      0.23        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.51      0.38      0.38       311\n",
            "weighted avg       0.59      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.603\tKappa: 0.084\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 26  68   1]\n",
            " [ 13 211   7]\n",
            " [  2  74  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.63      0.27      0.38        95\n",
            "      medium       0.60      0.91      0.72       231\n",
            "        high       0.62      0.15      0.24        89\n",
            "\n",
            "    accuracy                           0.60       415\n",
            "   macro avg       0.62      0.44      0.45       415\n",
            "weighted avg       0.61      0.60      0.54       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.694\tKappa: 0.194\tAccuracy: 0.602\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 16  54   2]\n",
            " [ 13 144   4]\n",
            " [  2  46  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.22      0.31        72\n",
            "      medium       0.59      0.89      0.71       161\n",
            "        high       0.74      0.26      0.39        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.62      0.46      0.47       298\n",
            "weighted avg       0.60      0.59      0.54       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.739\tKappa: 0.213\tAccuracy: 0.594\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  5  63   1]\n",
            " [  3 143   4]\n",
            " [  0  49  16]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.07      0.13        69\n",
            "      medium       0.56      0.95      0.71       150\n",
            "        high       0.76      0.25      0.37        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.65      0.42      0.40       284\n",
            "weighted avg       0.62      0.58      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.672\tKappa: 0.158\tAccuracy: 0.577\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.677\t Average Kappa: 0.162\t Average Accuracy: 0.605\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mzuUzKsDrKrY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_27_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer(ngram_range=(2, 2))),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', RFC(random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t7cWHdJtvukc",
        "colab_type": "code",
        "outputId": "c3e0f5a9-2486-44fb-99e4-6cc7a21c9ac2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_27_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  8 189   7]\n",
            " [  0  47   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.08      0.12        53\n",
            "      medium       0.66      0.93      0.77       204\n",
            "        high       0.50      0.13      0.21        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.50      0.38      0.37       311\n",
            "weighted avg       0.58      0.64      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.583\tKappa: 0.072\tAccuracy: 0.643\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 21  73   1]\n",
            " [ 17 207   7]\n",
            " [  1  76  12]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.54      0.22      0.31        95\n",
            "      medium       0.58      0.90      0.71       231\n",
            "        high       0.60      0.13      0.22        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.57      0.42      0.41       415\n",
            "weighted avg       0.58      0.58      0.51       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.669\tKappa: 0.141\tAccuracy: 0.578\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 27  44   1]\n",
            " [ 33 124   4]\n",
            " [  8  43  14]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.40      0.38      0.39        72\n",
            "      medium       0.59      0.77      0.67       161\n",
            "        high       0.74      0.22      0.33        65\n",
            "\n",
            "    accuracy                           0.55       298\n",
            "   macro avg       0.57      0.45      0.46       298\n",
            "weighted avg       0.57      0.55      0.53       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.689\tKappa: 0.186\tAccuracy: 0.554\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  6  62   1]\n",
            " [  6 137   7]\n",
            " [  3  55   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.40      0.09      0.14        69\n",
            "      medium       0.54      0.91      0.68       150\n",
            "        high       0.47      0.11      0.18        65\n",
            "\n",
            "    accuracy                           0.53       284\n",
            "   macro avg       0.47      0.37      0.33       284\n",
            "weighted avg       0.49      0.53      0.43       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.586\tKappa: 0.061\tAccuracy: 0.528\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.632\t Average Kappa: 0.115\t Average Accuracy: 0.576\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KIVYrxEi0LU",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que al combinar los unigramas originales (es decir, utilizados en los modelos anteriores) con bigramas el desempeño se ve perjudicado consistentemente. \n",
        "\n",
        "Además, como cabría de esperar, al utilizar solamente bigramas se pierde mucha información relevante y, por ende, el desempeño de los modelos empeora."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FAsHqEnTi0fl",
        "colab_type": "text"
      },
      "source": [
        "Siguiendo la línea de estudiar el desempeño de otros modelos de clasificación, en la siguiente sección se estudia el desempeño de Máquinas de Soporte Vectorial en el presente problema y con los features estudiados."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LNK8Wvski1ED",
        "colab_type": "text"
      },
      "source": [
        "### 5.11. Pruebas en Máquinas de Soporte Vectorial"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tp6Z7BSZ0ETe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_28_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', SVC(probability = True))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qtAdjclj1Kct",
        "colab_type": "code",
        "outputId": "874a5ef5-a4f7-406e-d7cf-a54c486747c3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_28_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  2  51   0]\n",
            " [  1 191  12]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.67      0.04      0.07        53\n",
            "      medium       0.66      0.94      0.78       204\n",
            "        high       0.40      0.15      0.22        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.58      0.37      0.35       311\n",
            "weighted avg       0.62      0.65      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.579\tKappa: 0.069\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 14  77   4]\n",
            " [  8 209  14]\n",
            " [  0  65  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.64      0.15      0.24        95\n",
            "      medium       0.60      0.90      0.72       231\n",
            "        high       0.57      0.27      0.37        89\n",
            "\n",
            "    accuracy                           0.60       415\n",
            "   macro avg       0.60      0.44      0.44       415\n",
            "weighted avg       0.60      0.60      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.658\tKappa: 0.183\tAccuracy: 0.595\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 10  57   5]\n",
            " [  3 141  17]\n",
            " [  0  35  30]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.77      0.14      0.24        72\n",
            "      medium       0.61      0.88      0.72       161\n",
            "        high       0.58      0.46      0.51        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.65      0.49      0.49       298\n",
            "weighted avg       0.64      0.61      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.738\tKappa: 0.258\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[  7  57   5]\n",
            " [  5 130  15]\n",
            " [  3  40  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.47      0.10      0.17        69\n",
            "      medium       0.57      0.87      0.69       150\n",
            "        high       0.52      0.34      0.41        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.52      0.44      0.42       284\n",
            "weighted avg       0.54      0.56      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.662\tKappa: 0.171\tAccuracy: 0.56\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.659\t Average Kappa: 0.17\t Average Accuracy: 0.602\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I3RBQ8lv0IB9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_29_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', SVC(probability = True))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6-ASY2CWD3bv",
        "colab_type": "code",
        "outputId": "19a15c79-a1a2-4795-a5ae-9adecacfb29b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_29_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  3 191  10]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.06      0.10        53\n",
            "      medium       0.67      0.94      0.78       204\n",
            "        high       0.44      0.15      0.22        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.54      0.38      0.37       311\n",
            "weighted avg       0.60      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.585\tKappa: 0.081\tAccuracy: 0.65\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 28  66   1]\n",
            " [ 14 194  23]\n",
            " [  2  61  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.64      0.29      0.40        95\n",
            "      medium       0.60      0.84      0.70       231\n",
            "        high       0.52      0.29      0.37        89\n",
            "\n",
            "    accuracy                           0.60       415\n",
            "   macro avg       0.59      0.48      0.49       415\n",
            "weighted avg       0.59      0.60      0.56       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.225\tAccuracy: 0.598\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 19  46   7]\n",
            " [ 19 124  18]\n",
            " [  2  28  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.47      0.26      0.34        72\n",
            "      medium       0.63      0.77      0.69       161\n",
            "        high       0.58      0.54      0.56        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.56      0.52      0.53       298\n",
            "weighted avg       0.58      0.60      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.728\tKappa: 0.287\tAccuracy: 0.597\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 14  51   4]\n",
            " [  8 124  18]\n",
            " [  3  41  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.20      0.30        69\n",
            "      medium       0.57      0.83      0.68       150\n",
            "        high       0.49      0.32      0.39        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.54      0.45      0.45       284\n",
            "weighted avg       0.55      0.56      0.52       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.682\tKappa: 0.188\tAccuracy: 0.56\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.666\t Average Kappa: 0.195\t Average Accuracy: 0.601\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mn3zaoWa0LL2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_30_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', SVC(probability = True))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1mOhPzdwEJL8",
        "colab_type": "code",
        "outputId": "0069a329-f426-4385-84f1-a7b9a1642e97",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_30_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  4 190  10]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.08      0.13        53\n",
            "      medium       0.67      0.93      0.78       204\n",
            "        high       0.44      0.15      0.22        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.54      0.38      0.38       311\n",
            "weighted avg       0.60      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.591\tKappa: 0.088\tAccuracy: 0.65\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 23  71   1]\n",
            " [ 12 194  25]\n",
            " [  1  63  25]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.64      0.24      0.35        95\n",
            "      medium       0.59      0.84      0.69       231\n",
            "        high       0.49      0.28      0.36        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.57      0.45      0.47       415\n",
            "weighted avg       0.58      0.58      0.54       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.189\tAccuracy: 0.583\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 22  44   6]\n",
            " [ 19 126  16]\n",
            " [  3  27  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.31      0.38        72\n",
            "      medium       0.64      0.78      0.70       161\n",
            "        high       0.61      0.54      0.57        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.58      0.54      0.55       298\n",
            "weighted avg       0.60      0.61      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.734\tKappa: 0.318\tAccuracy: 0.614\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 11  57   1]\n",
            " [  7 127  16]\n",
            " [  3  35  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.16      0.24        69\n",
            "      medium       0.58      0.85      0.69       150\n",
            "        high       0.61      0.42      0.50        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.57      0.47      0.48       284\n",
            "weighted avg       0.57      0.58      0.54       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.68\tKappa: 0.223\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.668\t Average Kappa: 0.204\t Average Accuracy: 0.607\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l8Tqw66-0Seu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_31_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', SVC(probability = True))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QCo4rbVMGgBR",
        "colab_type": "code",
        "outputId": "272e7d0a-bb24-4264-b4c5-198e3ff75b42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_31_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  4 189  11]\n",
            " [  0  47   7]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.08      0.13        53\n",
            "      medium       0.66      0.93      0.77       204\n",
            "        high       0.39      0.13      0.19        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.52      0.38      0.37       311\n",
            "weighted avg       0.59      0.64      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.59\tKappa: 0.072\tAccuracy: 0.643\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 25  69   1]\n",
            " [ 13 192  26]\n",
            " [  2  61  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.26      0.37        95\n",
            "      medium       0.60      0.83      0.69       231\n",
            "        high       0.49      0.29      0.37        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.57      0.46      0.48       415\n",
            "weighted avg       0.58      0.59      0.55       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.201\tAccuracy: 0.586\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 22  43   7]\n",
            " [ 19 126  16]\n",
            " [  3  26  36]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.31      0.38        72\n",
            "      medium       0.65      0.78      0.71       161\n",
            "        high       0.61      0.55      0.58        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.59      0.55      0.56       298\n",
            "weighted avg       0.60      0.62      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.734\tKappa: 0.326\tAccuracy: 0.617\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 12  56   1]\n",
            " [  7 128  15]\n",
            " [  4  39  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.17      0.26        69\n",
            "      medium       0.57      0.85      0.69       150\n",
            "        high       0.58      0.34      0.43        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.56      0.46      0.46       284\n",
            "weighted avg       0.56      0.57      0.52       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.678\tKappa: 0.197\tAccuracy: 0.57\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.667\t Average Kappa: 0.199\t Average Accuracy: 0.604\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQuAhVFDi0ul",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que, pese a que las dos últimas pruebas son bastante competitivas con los resultados obtenidos con Random Forest, en general una Máquina de Soporte Vectorial no ofrece una ganancia importante de desempeño, al menos en el problema estudiado.\n",
        "\n",
        "Notar que aquellos casos en que el desempeño es comparable a los más altos obtenidos anteriormente se debe principalmente a que la performance sobre los sentimientos `'Fear'` y `'Joy'` incrementan, pero disminuyen considerablemente en `'Anger'` y `'Sadness'`. Es decir que el hecho de promediar camufla la pérdida de desempeño, compensando con fortalecer modelos para sentimientos específicos."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hx0RRduuNsRq",
        "colab_type": "text"
      },
      "source": [
        "A continuación se estudiarán dos clasificadores que podrían explotar las relaciones espaciales entre los features encontrados. Estos son K-Nearest Neighbors y Perceptrón Multicapa (apelando a su similitud/equivalencia con los modelos lineales)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8R_yrf3Ai06I",
        "colab_type": "text"
      },
      "source": [
        "### 5.12. Pruebas con K-Nearest Neighbors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vQ-dJ61mOq-Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_32_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', KNC())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IueM5FAsPb-A",
        "colab_type": "code",
        "outputId": "98451371-cf3d-42b3-af58-baccc7721748",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_32_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  7  26  20]\n",
            " [ 15 130  59]\n",
            " [  7  36  11]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.24      0.13      0.17        53\n",
            "      medium       0.68      0.64      0.66       204\n",
            "        high       0.12      0.20      0.15        54\n",
            "\n",
            "    accuracy                           0.48       311\n",
            "   macro avg       0.35      0.32      0.33       311\n",
            "weighted avg       0.51      0.48      0.49       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.513\tKappa: 0.009\tAccuracy: 0.476\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 38  51   6]\n",
            " [ 64 148  19]\n",
            " [ 21  52  16]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.31      0.40      0.35        95\n",
            "      medium       0.59      0.64      0.61       231\n",
            "        high       0.39      0.18      0.25        89\n",
            "\n",
            "    accuracy                           0.49       415\n",
            "   macro avg       0.43      0.41      0.40       415\n",
            "weighted avg       0.48      0.49      0.47       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.569\tKappa: 0.106\tAccuracy: 0.487\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 25  41   6]\n",
            " [ 30 118  13]\n",
            " [ 11  37  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.38      0.35      0.36        72\n",
            "      medium       0.60      0.73      0.66       161\n",
            "        high       0.47      0.26      0.34        65\n",
            "\n",
            "    accuracy                           0.54       298\n",
            "   macro avg       0.48      0.45      0.45       298\n",
            "weighted avg       0.52      0.54      0.52       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.638\tKappa: 0.18\tAccuracy: 0.537\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 14  12  43]\n",
            " [ 13  27 110]\n",
            " [  8   7  50]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.40      0.20      0.27        69\n",
            "      medium       0.59      0.18      0.28       150\n",
            "        high       0.25      0.77      0.37        65\n",
            "\n",
            "    accuracy                           0.32       284\n",
            "   macro avg       0.41      0.38      0.31       284\n",
            "weighted avg       0.46      0.32      0.30       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.54\tKappa: 0.057\tAccuracy: 0.32\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.565\t Average Kappa: 0.088\t Average Accuracy: 0.455\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zHoGDXBJOtrA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_33_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', KNC())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D6a11ExnUEwE",
        "colab_type": "code",
        "outputId": "8898c0be-12b1-460f-a9a8-0598a3f401b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_33_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  8  36   9]\n",
            " [ 23 133  48]\n",
            " [  6  35  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.22      0.15      0.18        53\n",
            "      medium       0.65      0.65      0.65       204\n",
            "        high       0.19      0.24      0.21        54\n",
            "\n",
            "    accuracy                           0.50       311\n",
            "   macro avg       0.35      0.35      0.35       311\n",
            "weighted avg       0.50      0.50      0.49       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.513\tKappa: 0.011\tAccuracy: 0.495\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 37  49   9]\n",
            " [ 56 154  21]\n",
            " [ 15  59  15]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.34      0.39      0.36        95\n",
            "      medium       0.59      0.67      0.62       231\n",
            "        high       0.33      0.17      0.22        89\n",
            "\n",
            "    accuracy                           0.50       415\n",
            "   macro avg       0.42      0.41      0.40       415\n",
            "weighted avg       0.48      0.50      0.48       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.599\tKappa: 0.11\tAccuracy: 0.496\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 20  44   8]\n",
            " [ 29 119  13]\n",
            " [  4  38  23]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.38      0.28      0.32        72\n",
            "      medium       0.59      0.74      0.66       161\n",
            "        high       0.52      0.35      0.42        65\n",
            "\n",
            "    accuracy                           0.54       298\n",
            "   macro avg       0.50      0.46      0.47       298\n",
            "weighted avg       0.53      0.54      0.52       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.639\tKappa: 0.186\tAccuracy: 0.544\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 15  12  42]\n",
            " [ 19  25 106]\n",
            " [  7   6  52]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.37      0.22      0.27        69\n",
            "      medium       0.58      0.17      0.26       150\n",
            "        high       0.26      0.80      0.39        65\n",
            "\n",
            "    accuracy                           0.32       284\n",
            "   macro avg       0.40      0.39      0.31       284\n",
            "weighted avg       0.46      0.32      0.29       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.538\tKappa: 0.066\tAccuracy: 0.324\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.572\t Average Kappa: 0.0932\t Average Accuracy: 0.465\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3STYspyCO1kv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_34_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', KNC())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ddW2ysU5VSo9",
        "colab_type": "code",
        "outputId": "2a976267-167a-43c4-c254-0f9b44f0002b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_34_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  7  30  16]\n",
            " [ 21 146  37]\n",
            " [  7  34  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.20      0.13      0.16        53\n",
            "      medium       0.70      0.72      0.71       204\n",
            "        high       0.20      0.24      0.22        54\n",
            "\n",
            "    accuracy                           0.53       311\n",
            "   macro avg       0.36      0.36      0.36       311\n",
            "weighted avg       0.52      0.53      0.53       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.551\tKappa: 0.069\tAccuracy: 0.534\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 34  53   8]\n",
            " [ 60 152  19]\n",
            " [ 16  54  19]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.31      0.36      0.33        95\n",
            "      medium       0.59      0.66      0.62       231\n",
            "        high       0.41      0.21      0.28        89\n",
            "\n",
            "    accuracy                           0.49       415\n",
            "   macro avg       0.44      0.41      0.41       415\n",
            "weighted avg       0.49      0.49      0.48       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.582\tKappa: 0.109\tAccuracy: 0.494\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 22  43   7]\n",
            " [ 29 116  16]\n",
            " [  6  35  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.39      0.31      0.34        72\n",
            "      medium       0.60      0.72      0.65       161\n",
            "        high       0.51      0.37      0.43        65\n",
            "\n",
            "    accuracy                           0.54       298\n",
            "   macro avg       0.50      0.47      0.47       298\n",
            "weighted avg       0.53      0.54      0.53       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.642\tKappa: 0.196\tAccuracy: 0.544\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 17  11  41]\n",
            " [ 16  32 102]\n",
            " [  6  13  46]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.44      0.25      0.31        69\n",
            "      medium       0.57      0.21      0.31       150\n",
            "        high       0.24      0.71      0.36        65\n",
            "\n",
            "    accuracy                           0.33       284\n",
            "   macro avg       0.42      0.39      0.33       284\n",
            "weighted avg       0.46      0.33      0.32       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.548\tKappa: 0.063\tAccuracy: 0.335\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.581\t Average Kappa: 0.109\t Average Accuracy: 0.477\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iHGywtjjO1sx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_35_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', KNC())])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iMzEKfCwX_xf",
        "colab_type": "code",
        "outputId": "da20ef72-548c-4e00-f500-f3dec529ecd2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_35_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  8  26  19]\n",
            " [ 25 144  35]\n",
            " [  6  38  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.21      0.15      0.17        53\n",
            "      medium       0.69      0.71      0.70       204\n",
            "        high       0.16      0.19      0.17        54\n",
            "\n",
            "    accuracy                           0.52       311\n",
            "   macro avg       0.35      0.35      0.35       311\n",
            "weighted avg       0.52      0.52      0.52       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.547\tKappa: 0.05\tAccuracy: 0.521\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 32  58   5]\n",
            " [ 50 157  24]\n",
            " [ 15  57  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.34      0.33        95\n",
            "      medium       0.58      0.68      0.62       231\n",
            "        high       0.37      0.19      0.25        89\n",
            "\n",
            "    accuracy                           0.50       415\n",
            "   macro avg       0.43      0.40      0.40       415\n",
            "weighted avg       0.48      0.50      0.48       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.58\tKappa: 0.097\tAccuracy: 0.496\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 23  42   7]\n",
            " [ 30 116  15]\n",
            " [  6  39  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.39      0.32      0.35        72\n",
            "      medium       0.59      0.72      0.65       161\n",
            "        high       0.48      0.31      0.37        65\n",
            "\n",
            "    accuracy                           0.53       298\n",
            "   macro avg       0.48      0.45      0.46       298\n",
            "weighted avg       0.52      0.53      0.52       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.635\tKappa: 0.173\tAccuracy: 0.534\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[12 14 43]\n",
            " [20 34 96]\n",
            " [ 7 12 46]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.31      0.17      0.22        69\n",
            "      medium       0.57      0.23      0.32       150\n",
            "        high       0.25      0.71      0.37        65\n",
            "\n",
            "    accuracy                           0.32       284\n",
            "   macro avg       0.37      0.37      0.30       284\n",
            "weighted avg       0.43      0.32      0.31       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.557\tKappa: 0.042\tAccuracy: 0.324\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.58\t Average Kappa: 0.0905\t Average Accuracy: 0.469\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DwTCDMpvY4C9",
        "colab_type": "text"
      },
      "source": [
        "Se aprecia que en forma consistente el clasificador KNN ofrece una peor performance al utilizar los features obtenidos a partir de los datos. Una posible solución podría ser cambiar el modelo detrás de `CountVectorizer` pero en vista de que el desempeño general de los modelos anteriormente vistos es bastante superior se considera que se pueden alcanzar resultados competentes sin recurrir a esta acción."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "boHeZUr5OdL9",
        "colab_type": "text"
      },
      "source": [
        "### 5.13. Pruebas con Perceptrón Multicapa"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RAHiaKrkO-YF",
        "colab": {}
      },
      "source": [
        "def get_experiment_36_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler())\n",
        "                                    ])), ('clf', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nFQTv_HRdXao",
        "colab_type": "code",
        "outputId": "6066310b-ca46-4eae-ab16-bc5f2d299057",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_36_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  6 190   8]\n",
            " [  2  48   4]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.08      0.12        53\n",
            "      medium       0.66      0.93      0.77       204\n",
            "        high       0.33      0.07      0.12        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.44      0.36      0.34       311\n",
            "weighted avg       0.55      0.64      0.55       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.535\tKappa: 0.047\tAccuracy: 0.637\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[  6  85   4]\n",
            " [  4 213  14]\n",
            " [  0  81   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.06      0.11        95\n",
            "      medium       0.56      0.92      0.70       231\n",
            "        high       0.31      0.09      0.14        89\n",
            "\n",
            "    accuracy                           0.55       415\n",
            "   macro avg       0.49      0.36      0.32       415\n",
            "weighted avg       0.52      0.55      0.44       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.519\tKappa: 0.042\tAccuracy: 0.547\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[  6  62   4]\n",
            " [  5 141  15]\n",
            " [  2  41  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.46      0.08      0.14        72\n",
            "      medium       0.58      0.88      0.70       161\n",
            "        high       0.54      0.34      0.42        65\n",
            "\n",
            "    accuracy                           0.57       298\n",
            "   macro avg       0.53      0.43      0.42       298\n",
            "weighted avg       0.54      0.57      0.50       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.617\tKappa: 0.163\tAccuracy: 0.567\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 13  54   2]\n",
            " [ 14 132   4]\n",
            " [  1  63   1]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.46      0.19      0.27        69\n",
            "      medium       0.53      0.88      0.66       150\n",
            "        high       0.14      0.02      0.03        65\n",
            "\n",
            "    accuracy                           0.51       284\n",
            "   macro avg       0.38      0.36      0.32       284\n",
            "weighted avg       0.43      0.51      0.42       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.555\tKappa: 0.042\tAccuracy: 0.514\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.556\t Average Kappa: 0.0735\t Average Accuracy: 0.566\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "POtFQnZBO9QE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_37_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OpFmbUieslck",
        "colab_type": "code",
        "outputId": "2ebbab38-d692-4889-ce5d-c11140a666d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_37_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 15  36   2]\n",
            " [ 25 148  31]\n",
            " [  1  27  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.37      0.28      0.32        53\n",
            "      medium       0.70      0.73      0.71       204\n",
            "        high       0.44      0.48      0.46        54\n",
            "\n",
            "    accuracy                           0.61       311\n",
            "   macro avg       0.50      0.50      0.50       311\n",
            "weighted avg       0.60      0.61      0.60       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.639\tKappa: 0.215\tAccuracy: 0.608\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 33  59   3]\n",
            " [ 27 176  28]\n",
            " [  1  47  41]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.54      0.35      0.42        95\n",
            "      medium       0.62      0.76      0.69       231\n",
            "        high       0.57      0.46      0.51        89\n",
            "\n",
            "    accuracy                           0.60       415\n",
            "   macro avg       0.58      0.52      0.54       415\n",
            "weighted avg       0.59      0.60      0.59       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.681\tKappa: 0.278\tAccuracy: 0.602\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 26  43   3]\n",
            " [ 15 125  21]\n",
            " [  4  27  34]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.36      0.44        72\n",
            "      medium       0.64      0.78      0.70       161\n",
            "        high       0.59      0.52      0.55        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.60      0.55      0.57       298\n",
            "weighted avg       0.61      0.62      0.61       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.743\tKappa: 0.332\tAccuracy: 0.621\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[18 37 14]\n",
            " [29 89 32]\n",
            " [ 3 32 30]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.36      0.26      0.30        69\n",
            "      medium       0.56      0.59      0.58       150\n",
            "        high       0.39      0.46      0.43        65\n",
            "\n",
            "    accuracy                           0.48       284\n",
            "   macro avg       0.44      0.44      0.44       284\n",
            "weighted avg       0.48      0.48      0.48       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.633\tKappa: 0.14\tAccuracy: 0.482\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.674\t Average Kappa: 0.241\t Average Accuracy: 0.578\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "znAAnNfTPDH1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_38_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H2wlIu-1uCc9",
        "colab_type": "code",
        "outputId": "e43b1fda-3c9d-489f-c950-75a3955fcdba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_38_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 15  37   1]\n",
            " [ 22 159  23]\n",
            " [  1  33  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.39      0.28      0.33        53\n",
            "      medium       0.69      0.78      0.73       204\n",
            "        high       0.45      0.37      0.41        54\n",
            "\n",
            "    accuracy                           0.62       311\n",
            "   macro avg       0.51      0.48      0.49       311\n",
            "weighted avg       0.60      0.62      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.633\tKappa: 0.202\tAccuracy: 0.624\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 37  55   3]\n",
            " [ 39 163  29]\n",
            " [  4  44  41]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.46      0.39      0.42        95\n",
            "      medium       0.62      0.71      0.66       231\n",
            "        high       0.56      0.46      0.51        89\n",
            "\n",
            "    accuracy                           0.58       415\n",
            "   macro avg       0.55      0.52      0.53       415\n",
            "weighted avg       0.57      0.58      0.57       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.677\tKappa: 0.26\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 29  42   1]\n",
            " [ 25 121  15]\n",
            " [  5  29  31]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.49      0.40      0.44        72\n",
            "      medium       0.63      0.75      0.69       161\n",
            "        high       0.66      0.48      0.55        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.59      0.54      0.56       298\n",
            "weighted avg       0.60      0.61      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.753\tKappa: 0.311\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[18 42  9]\n",
            " [28 99 23]\n",
            " [ 6 31 28]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.35      0.26      0.30        69\n",
            "      medium       0.58      0.66      0.61       150\n",
            "        high       0.47      0.43      0.45        65\n",
            "\n",
            "    accuracy                           0.51       284\n",
            "   macro avg       0.46      0.45      0.45       284\n",
            "weighted avg       0.49      0.51      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.625\tKappa: 0.167\tAccuracy: 0.511\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.672\t Average Kappa: 0.235\t Average Accuracy: 0.581\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vLX8U03lPFBi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_39_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', ModifiedCharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False))),\n",
        "                                    ('emphasis', EmphasisHandler())\n",
        "                                    ])), ('clf', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NGTj2iepuEYr",
        "colab_type": "code",
        "outputId": "4d3d1553-4746-45ab-d902-65f39a8e516d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_39_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 17  35   1]\n",
            " [ 27 155  22]\n",
            " [  0  33  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.39      0.32      0.35        53\n",
            "      medium       0.70      0.76      0.73       204\n",
            "        high       0.48      0.39      0.43        54\n",
            "\n",
            "    accuracy                           0.62       311\n",
            "   macro avg       0.52      0.49      0.50       311\n",
            "weighted avg       0.60      0.62      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.64\tKappa: 0.211\tAccuracy: 0.621\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 38  51   6]\n",
            " [ 32 162  37]\n",
            " [  3  42  44]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.40      0.45        95\n",
            "      medium       0.64      0.70      0.67       231\n",
            "        high       0.51      0.49      0.50        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.55      0.53      0.54       415\n",
            "weighted avg       0.58      0.59      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.678\tKappa: 0.281\tAccuracy: 0.588\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 33  35   4]\n",
            " [ 36 104  21]\n",
            " [  6  24  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.44      0.46      0.45        72\n",
            "      medium       0.64      0.65      0.64       161\n",
            "        high       0.58      0.54      0.56        65\n",
            "\n",
            "    accuracy                           0.58       298\n",
            "   macro avg       0.55      0.55      0.55       298\n",
            "weighted avg       0.58      0.58      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.742\tKappa: 0.295\tAccuracy: 0.577\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[18 40 11]\n",
            " [26 97 27]\n",
            " [ 4 30 31]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.38      0.26      0.31        69\n",
            "      medium       0.58      0.65      0.61       150\n",
            "        high       0.45      0.48      0.46        65\n",
            "\n",
            "    accuracy                           0.51       284\n",
            "   macro avg       0.47      0.46      0.46       284\n",
            "weighted avg       0.50      0.51      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.629\tKappa: 0.18\tAccuracy: 0.514\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.672\t Average Kappa: 0.242\t Average Accuracy: 0.575\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sW9gxxLTO86y",
        "colab_type": "text"
      },
      "source": [
        "En esta oportunidad y por simplicidad no se muestran las pruebas realizadas sobre la forma de la arquitectura (número de capas y de neuronas por capa), por lo que solo se presentan las arquitecturas que presentaron un mejor desempeño y más balanceado.\n",
        "\n",
        "En términos generales, el desempeño de las redes es comparable al obtenido por medio de Random Forest (salvo en el primer experimento presentado en la presente sección). De hecho, el criterio Kappa mejora considerablemente con esta arquitectura, aunque en desmedro de las otras dos métricas."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2-ApeO8JGmmX",
        "colab_type": "text"
      },
      "source": [
        "### 5.14. Voting Classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sl8CvMr0GvEZ",
        "colab_type": "text"
      },
      "source": [
        "Dada la clara superioridad del clasificador presentado en `get_experiment_17_pipeline` respecto a las demás pruebas mostradas, salvo en cuando a la métrica Kappa donde se vió superado por la Red Neuronal presente en `get_experiment_37_pipeline`, en la presente sección se propone ajustar un poco los parámetros del primer clasificador, para luego combinar estas dos estrategias por medio de un ensamble basado en Voting Classifier.\n",
        "\n",
        "Dado que se requiere que nuestro modelo pueda implementar el método `predict_proba` no es posible emplear el parámetro `'hard'` para el método de votación (que corresponde al voto de la mayoría), por lo que imperantemente se debe emplear `'soft'` voting, es decir el promedio."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5cozg46OKg2x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_40_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VKqTsgegKq_D",
        "colab_type": "code",
        "outputId": "fde95855-6ebf-4469-d421-6a9eb4a03598",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_40_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  7 188   9]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.30      0.06      0.10        53\n",
            "      medium       0.66      0.92      0.77       204\n",
            "        high       0.47      0.15      0.23        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.48      0.38      0.36       311\n",
            "weighted avg       0.57      0.64      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.648\tKappa: 0.067\tAccuracy: 0.64\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 30  64   1]\n",
            " [ 15 208   8]\n",
            " [  0  68  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.67      0.32      0.43        95\n",
            "      medium       0.61      0.90      0.73       231\n",
            "        high       0.70      0.24      0.35        89\n",
            "\n",
            "    accuracy                           0.62       415\n",
            "   macro avg       0.66      0.48      0.50       415\n",
            "weighted avg       0.64      0.62      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.707\tKappa: 0.254\tAccuracy: 0.624\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 23  47   2]\n",
            " [ 13 140   8]\n",
            " [  2  36  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.61      0.32      0.42        72\n",
            "      medium       0.63      0.87      0.73       161\n",
            "        high       0.73      0.42      0.53        65\n",
            "\n",
            "    accuracy                           0.64       298\n",
            "   macro avg       0.65      0.53      0.56       298\n",
            "weighted avg       0.64      0.64      0.61       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.738\tKappa: 0.326\tAccuracy: 0.638\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 12  55   2]\n",
            " [  4 134  12]\n",
            " [  0  41  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.75      0.17      0.28        69\n",
            "      medium       0.58      0.89      0.71       150\n",
            "        high       0.63      0.37      0.47        65\n",
            "\n",
            "    accuracy                           0.60       284\n",
            "   macro avg       0.65      0.48      0.48       284\n",
            "weighted avg       0.63      0.60      0.55       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.686\tKappa: 0.24\tAccuracy: 0.599\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.695\t Average Kappa: 0.222\t Average Accuracy: 0.625\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eH9bXZWfLLID",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_41_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(n_estimators=300, random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "phYvh3ffLPiL",
        "colab_type": "code",
        "outputId": "8a08d625-ac94-4d27-9538-f5a375ec5cc1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_41_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  7 188   9]\n",
            " [  0  45   9]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.30      0.06      0.10        53\n",
            "      medium       0.66      0.92      0.77       204\n",
            "        high       0.50      0.17      0.25        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.49      0.38      0.37       311\n",
            "weighted avg       0.57      0.64      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.643\tKappa: 0.079\tAccuracy: 0.643\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 29  65   1]\n",
            " [ 15 207   9]\n",
            " [  1  66  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.64      0.31      0.41        95\n",
            "      medium       0.61      0.90      0.73       231\n",
            "        high       0.69      0.25      0.36        89\n",
            "\n",
            "    accuracy                           0.62       415\n",
            "   macro avg       0.65      0.48      0.50       415\n",
            "weighted avg       0.64      0.62      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.71\tKappa: 0.251\tAccuracy: 0.622\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 22  48   2]\n",
            " [ 13 140   8]\n",
            " [  2  38  25]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.31      0.40        72\n",
            "      medium       0.62      0.87      0.72       161\n",
            "        high       0.71      0.38      0.50        65\n",
            "\n",
            "    accuracy                           0.63       298\n",
            "   macro avg       0.64      0.52      0.54       298\n",
            "weighted avg       0.63      0.63      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.74\tKappa: 0.303\tAccuracy: 0.628\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 13  54   2]\n",
            " [  4 134  12]\n",
            " [  1  41  23]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.72      0.19      0.30        69\n",
            "      medium       0.59      0.89      0.71       150\n",
            "        high       0.62      0.35      0.45        65\n",
            "\n",
            "    accuracy                           0.60       284\n",
            "   macro avg       0.64      0.48      0.49       284\n",
            "weighted avg       0.63      0.60      0.55       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.686\tKappa: 0.241\tAccuracy: 0.599\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.695\t Average Kappa: 0.218\t Average Accuracy: 0.623\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9188CJltLZbl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_42_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', RFC(n_estimators=75, random_state = 1138, class_weight = 'balanced'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bR81_xe8Ld3T",
        "colab_type": "code",
        "outputId": "4f576d90-ba4a-432a-ad3b-42e188ca8162",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_42_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  3  50   0]\n",
            " [  9 189   6]\n",
            " [  0  45   9]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.25      0.06      0.09        53\n",
            "      medium       0.67      0.93      0.77       204\n",
            "        high       0.60      0.17      0.26        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.51      0.38      0.38       311\n",
            "weighted avg       0.58      0.65      0.57       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.645\tKappa: 0.084\tAccuracy: 0.646\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 29  65   1]\n",
            " [ 15 207   9]\n",
            " [  1  66  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.64      0.31      0.41        95\n",
            "      medium       0.61      0.90      0.73       231\n",
            "        high       0.69      0.25      0.36        89\n",
            "\n",
            "    accuracy                           0.62       415\n",
            "   macro avg       0.65      0.48      0.50       415\n",
            "weighted avg       0.64      0.62      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.707\tKappa: 0.251\tAccuracy: 0.622\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 18  52   2]\n",
            " [ 15 136  10]\n",
            " [  2  39  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.25      0.34        72\n",
            "      medium       0.60      0.84      0.70       161\n",
            "        high       0.67      0.37      0.48        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.49      0.50       298\n",
            "weighted avg       0.59      0.60      0.56       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.727\tKappa: 0.246\tAccuracy: 0.597\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  57   2]\n",
            " [  6 132  12]\n",
            " [  1  41  23]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.14      0.23        69\n",
            "      medium       0.57      0.88      0.69       150\n",
            "        high       0.62      0.35      0.45        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.59      0.46      0.46       284\n",
            "weighted avg       0.59      0.58      0.53       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.682\tKappa: 0.206\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.69\t Average Kappa: 0.197\t Average Accuracy: 0.611\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x3hHMQVmK_sU",
        "colab_type": "text"
      },
      "source": [
        "Incrementar a 200 el número de estimadores en Random Forest ofrece una mejor performance, mientras que al aumentar aún más esta cantidad comienza a deteriorar el resultado. Por otro lado, esto último también ocurre al reducir la cantidad de estimadores a 75.\n",
        "\n",
        "Por lo tanto, se escoge fijar este número en 200.\n",
        "\n",
        "Ahora, se procede a evaluar el Clasificador por Votación."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "czkp728kMIrn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_43_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138)),\n",
        "                                                                   ('MLP2', MLP(hidden_layer_sizes=(2000, 500, 100, )))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B9peiVMaNbip",
        "colab_type": "code",
        "outputId": "0614f17a-6e93-411a-8f7f-6964abbf7b1d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_43_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 11  42   0]\n",
            " [ 18 169  17]\n",
            " [  1  35  18]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.37      0.21      0.27        53\n",
            "      medium       0.69      0.83      0.75       204\n",
            "        high       0.51      0.33      0.40        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.52      0.46      0.47       311\n",
            "weighted avg       0.60      0.64      0.61       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.646\tKappa: 0.184\tAccuracy: 0.637\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 30  63   2]\n",
            " [ 26 181  24]\n",
            " [  2  52  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.52      0.32      0.39        95\n",
            "      medium       0.61      0.78      0.69       231\n",
            "        high       0.57      0.39      0.47        89\n",
            "\n",
            "    accuracy                           0.59       415\n",
            "   macro avg       0.57      0.50      0.52       415\n",
            "weighted avg       0.58      0.59      0.57       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.695\tKappa: 0.245\tAccuracy: 0.593\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 25  44   3]\n",
            " [ 13 131  17]\n",
            " [  3  28  34]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.61      0.35      0.44        72\n",
            "      medium       0.65      0.81      0.72       161\n",
            "        high       0.63      0.52      0.57        65\n",
            "\n",
            "    accuracy                           0.64       298\n",
            "   macro avg       0.63      0.56      0.58       298\n",
            "weighted avg       0.63      0.64      0.62       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.75\tKappa: 0.352\tAccuracy: 0.638\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[17 40 12]\n",
            " [23 99 28]\n",
            " [ 3 34 28]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.40      0.25      0.30        69\n",
            "      medium       0.57      0.66      0.61       150\n",
            "        high       0.41      0.43      0.42        65\n",
            "\n",
            "    accuracy                           0.51       284\n",
            "   macro avg       0.46      0.45      0.45       284\n",
            "weighted avg       0.49      0.51      0.49       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.676\tKappa: 0.16\tAccuracy: 0.507\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.692\t Average Kappa: 0.235\t Average Accuracy: 0.594\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OZeciZnmRA7C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_44_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vJb1XTirRwyq",
        "colab_type": "code",
        "outputId": "9f1ee275-59fc-4e7d-d37b-965f48be5582",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_44_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  8  45   0]\n",
            " [ 15 174  15]\n",
            " [  0  41  13]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.35      0.15      0.21        53\n",
            "      medium       0.67      0.85      0.75       204\n",
            "        high       0.46      0.24      0.32        54\n",
            "\n",
            "    accuracy                           0.63       311\n",
            "   macro avg       0.49      0.41      0.43       311\n",
            "weighted avg       0.58      0.63      0.58       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.647\tKappa: 0.119\tAccuracy: 0.627\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 28  66   1]\n",
            " [ 19 195  17]\n",
            " [  1  57  31]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.29      0.39        95\n",
            "      medium       0.61      0.84      0.71       231\n",
            "        high       0.63      0.35      0.45        89\n",
            "\n",
            "    accuracy                           0.61       415\n",
            "   macro avg       0.61      0.50      0.52       415\n",
            "weighted avg       0.61      0.61      0.58       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.709\tKappa: 0.256\tAccuracy: 0.612\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 20  49   3]\n",
            " [ 12 136  13]\n",
            " [  3  27  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.28      0.37        72\n",
            "      medium       0.64      0.84      0.73       161\n",
            "        high       0.69      0.54      0.60        65\n",
            "\n",
            "    accuracy                           0.64       298\n",
            "   macro avg       0.63      0.55      0.57       298\n",
            "weighted avg       0.63      0.64      0.62       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.752\tKappa: 0.347\tAccuracy: 0.641\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 18  43   8]\n",
            " [ 14 113  23]\n",
            " [  1  36  28]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.55      0.26      0.35        69\n",
            "      medium       0.59      0.75      0.66       150\n",
            "        high       0.47      0.43      0.45        65\n",
            "\n",
            "    accuracy                           0.56       284\n",
            "   macro avg       0.54      0.48      0.49       284\n",
            "weighted avg       0.55      0.56      0.54       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.676\tKappa: 0.224\tAccuracy: 0.56\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.696\t Average Kappa: 0.236\t Average Accuracy: 0.61\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3mKjo5EeRCLQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_45_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced'))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3X-wX6a7Tg0O",
        "colab_type": "code",
        "outputId": "66c0169c-bd55-4eac-9d02-ead2700adec2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_45_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  5  48   0]\n",
            " [  8 187   9]\n",
            " [  0  44  10]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.38      0.09      0.15        53\n",
            "      medium       0.67      0.92      0.77       204\n",
            "        high       0.53      0.19      0.27        54\n",
            "\n",
            "    accuracy                           0.65       311\n",
            "   macro avg       0.53      0.40      0.40       311\n",
            "weighted avg       0.60      0.65      0.58       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.64\tKappa: 0.11\tAccuracy: 0.65\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 31  64   0]\n",
            " [ 17 206   8]\n",
            " [  0  71  18]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.33      0.43        95\n",
            "      medium       0.60      0.89      0.72       231\n",
            "        high       0.69      0.20      0.31        89\n",
            "\n",
            "    accuracy                           0.61       415\n",
            "   macro avg       0.65      0.47      0.49       415\n",
            "weighted avg       0.63      0.61      0.57       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.708\tKappa: 0.233\tAccuracy: 0.614\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 21  49   2]\n",
            " [ 16 138   7]\n",
            " [  3  38  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.29      0.38        72\n",
            "      medium       0.61      0.86      0.72       161\n",
            "        high       0.73      0.37      0.49        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.62      0.51      0.53       298\n",
            "weighted avg       0.62      0.61      0.58       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.731\tKappa: 0.279\tAccuracy: 0.614\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 11  56   2]\n",
            " [  3 134  13]\n",
            " [  0  43  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.79      0.16      0.27        69\n",
            "      medium       0.58      0.89      0.70       150\n",
            "        high       0.59      0.34      0.43        65\n",
            "\n",
            "    accuracy                           0.59       284\n",
            "   macro avg       0.65      0.46      0.47       284\n",
            "weighted avg       0.63      0.59      0.53       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.694\tKappa: 0.215\tAccuracy: 0.588\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.693\t Average Kappa: 0.209\t Average Accuracy: 0.617\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LhVl7kt8RFd3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_46_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC4', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC5', RFC(n_estimators=200, class_weight = 'balanced'))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tq5iKjTpUATr",
        "colab_type": "code",
        "outputId": "24f2a9a8-c130-49d5-b2a6-054cedd1135a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_46_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  4  49   0]\n",
            " [  8 187   9]\n",
            " [  0  46   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.08      0.12        53\n",
            "      medium       0.66      0.92      0.77       204\n",
            "        high       0.47      0.15      0.23        54\n",
            "\n",
            "    accuracy                           0.64       311\n",
            "   macro avg       0.49      0.38      0.37       311\n",
            "weighted avg       0.57      0.64      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.635\tKappa: 0.075\tAccuracy: 0.64\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 29  65   1]\n",
            " [ 15 208   8]\n",
            " [  0  69  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.66      0.31      0.42        95\n",
            "      medium       0.61      0.90      0.73       231\n",
            "        high       0.69      0.22      0.34        89\n",
            "\n",
            "    accuracy                           0.62       415\n",
            "   macro avg       0.65      0.48      0.49       415\n",
            "weighted avg       0.64      0.62      0.57       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.711\tKappa: 0.242\tAccuracy: 0.619\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 23  48   1]\n",
            " [ 14 141   6]\n",
            " [  2  39  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.32      0.41        72\n",
            "      medium       0.62      0.88      0.72       161\n",
            "        high       0.77      0.37      0.50        65\n",
            "\n",
            "    accuracy                           0.63       298\n",
            "   macro avg       0.66      0.52      0.55       298\n",
            "weighted avg       0.65      0.63      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.731\tKappa: 0.307\tAccuracy: 0.631\n",
            "------------------------------------------------------\n",
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 10  58   1]\n",
            " [  4 133  13]\n",
            " [  0  43  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.71      0.14      0.24        69\n",
            "      medium       0.57      0.89      0.69       150\n",
            "        high       0.61      0.34      0.44        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.63      0.46      0.46       284\n",
            "weighted avg       0.61      0.58      0.52       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.693\tKappa: 0.2\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.693\t Average Kappa: 0.206\t Average Accuracy: 0.618\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtRH2UbtWB2C",
        "colab_type": "text"
      },
      "source": [
        "De los experimentos realizados en la presente sección, se concluye que el que ofrece resultados más prometedores es la configuración con 3 clasificadores Random Forest y una red MLP. \n",
        "\n",
        "Si bien los últimos dos experimentos son bastante sólidos, el buen desempeño de estos radica en mejorar los clasificadores para los sentimientos de `'fear'` y `'joy'`, perjudicando los demás. Por lo tanto, en base a la consistencia, se escoge como el mejor a la configuración ya mencionada."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8s-CJfCMV7zW",
        "colab_type": "text"
      },
      "source": [
        "### 5.15 Word Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eR08jzLYV-3P",
        "colab_type": "text"
      },
      "source": [
        "En esta sección se estudia reemplazar el extractor `CountVectorizer` del modelo escogido por una red de Word Embedding basada en algún modelo. Este modelo se escoge de entre las arquitecturas BERT y ELMo.\n",
        "\n",
        "Por lo tanto se realizan pruebas sobre ambas arquitectura y la que se considere que entrega mejores resultados será incorporada al proceso de extracción de características del modelo escogido en 5.14. Para verificar que dicho modelo es el indicado considerando el nuevo vector de características se realizan algunas pruebas sobre la estructura del Voting Classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mbi-tok6YIuD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tqdm.notebook import tqdm\n",
        "from bs4 import BeautifulSoup\n",
        "from pycontractions import Contractions\n",
        "from allennlp.commands.elmo import ElmoEmbedder\n",
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from transformers import BertTokenizer, BertForSequenceClassification, AdamW, BertConfig, get_linear_schedule_with_warmup\n",
        "from torch.optim.lr_scheduler import LambdaLR"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lMQIkSVwX5Ad",
        "colab_type": "code",
        "outputId": "fd1f2d0d-b088-44d1-a8a1-84ae61eb1cfc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        }
      },
      "source": [
        "### Load models\n",
        "# ELMo Embedder\n",
        "ee = ElmoEmbedder(options_file='https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json', \n",
        "                               weight_file='https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_weights.hdf5')\n",
        "# Contractions converter\n",
        "cont = Contractions(api_key=\"glove-twitter-100\")\n",
        "cont.load_models()\n",
        "# Bert Tokenizer\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n",
        "\n",
        "### Select device\n",
        "if torch.cuda.is_available():       \n",
        "    device = torch.device(\"cuda\")\n",
        "else:\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:253: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8C0YIzIiX5sY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### Utils\n",
        "def tweet_clean(tweet):\n",
        "    '''Tweet cleaner function'''\n",
        "    # Split in separator chars\n",
        "    tweet = tweet.rstrip()\n",
        "    # Remove html tags\n",
        "    tweet = BeautifulSoup(tweet).get_text()\n",
        "    # Remove mentions (@user)\n",
        "    tweet = ' '.join(re.sub(\"(@[A-Za-z0-9_]+)\", \" \", tweet).split())\n",
        "    # Remove urls\n",
        "    tweet = ' '.join(re.sub(\"(\\w+:\\/\\/\\S+)\", \" \", tweet).split())\n",
        "    # Replace emojis with descriptive text\n",
        "    tweet = emoji.demojize(tweet)\n",
        "    # Stemming [VALE PICO]\n",
        "    #tweet = ' '.join([stemmer.stem(word) for word in tweet.split()])\n",
        "    # Remove special chars except ! and ?\n",
        "    for char in \"_-/\\:,.#$%&()*+/:;<=>@[\\\\]^{|}~’`''\":\n",
        "        tweet = ' '.join(tweet.split(char))\n",
        "    # Separate ! and ? chars\n",
        "    tweet = tweet.replace(\"!\", \" ! \")\n",
        "    tweet = tweet.replace(\"?\", \" ? \")\n",
        "    # Acronyms translation\n",
        "    tweet = ' '.join([acronyms_dict.get(word.upper(), word) for word in tweet.split()])\n",
        "    # Remove numbers\n",
        "    tweet = ''.join([char for char in tweet if not char.isdigit()])\n",
        "    # Remove 's\n",
        "    tweet = ' '.join([word.strip() for word in tweet.split() if word != \"s\"])\n",
        "    # Lower case\n",
        "    tweet = tweet.lower()\n",
        "    return tweet\n",
        "\n",
        "# Acronyms dictionary\n",
        "acronyms_dict = {\"AFAIK\": \"As Far As I Know\",\n",
        "                \"AFK\": \"Away From Keyboard\",\n",
        "                \"ASAP\": \"As Soon As Possible\",\n",
        "                \"ATK\": \"At The Keyboard\",\n",
        "                \"ATM\": \"At The Moment\",\n",
        "                \"A3\": \"Anytime, Anywhere, Anyplace\",\n",
        "                \"BAK\": \"Back At Keyboard\",\n",
        "                \"BBL\": \"Be Back Later\",\n",
        "                \"BBS\": \"Be Back Soon\",\n",
        "                \"BFN\": \"Bye For Now\",\n",
        "                \"B4N\": \"Bye For Now\",\n",
        "                \"BRB\": \"Be Right Back\",\n",
        "                \"BRT\": \"Be Right There\",\n",
        "                \"BTW\": \"By The Way\",\n",
        "                \"B4\": \"Before\",\n",
        "                \"B4N\": \"Bye For Now\",\n",
        "                \"CU\": \"See You\",\n",
        "                \"CUL8R\": \"See You Later\",\n",
        "                \"CYA\": \"See You\",\n",
        "                \"FAQ\": \"Frequently Asked Questions\",\n",
        "                \"FC\": \"Fingers Crossed\",\n",
        "                \"FWIW\": \"For What It is Worth\",\n",
        "                \"FYI\": \"For Your Information\",\n",
        "                \"GAL\": \"Get A Life\",\n",
        "                \"GG\": \"Good Game\",\n",
        "                \"GN\": \"Good Night\",\n",
        "                \"GMTA\":\"Great Minds Think Alike\",\n",
        "                \"GR8\": \"Great!\",\n",
        "                \"G9\": \"Genius\",\n",
        "                \"IC\": \"I See\",\n",
        "                \"ICQ\":\"I Seek you\",\n",
        "                \"ILU\": \"I Love You\",\n",
        "                \"IMHO\": \"In My Humble Opinion\",\n",
        "                \"IMO\": \"In My Opinion\",\n",
        "                \"IOW\": \"In Other Words\",\n",
        "                \"IRL\": \"In Real Life\",\n",
        "                #\"KISS\": \"Keep It Simple, Stupid\",\n",
        "                \"LDR\": \"Long Distance Relationship\",\n",
        "                \"LMAO\": \"Laugh My Ass Off\",\n",
        "                \"LOL\": \"Laughing Out Loud\",\n",
        "                \"LTNS\": \"Long Time No See\",\n",
        "                \"L8R\": \"Later\",\n",
        "                \"MTE\": \"My Thoughts Exactly\",\n",
        "                \"M8\": \"Mate\",\n",
        "                \"NRN\": \"No Reply Necessary\",\n",
        "                \"OIC\": \"Oh I See\",\n",
        "                \"PITA\": \"Pain In The Ass\",\n",
        "                \"PRT\": \"Party\",\n",
        "                \"PRW\": \"Parents Are Watching\",\n",
        "                #\"QPSA?\":\t\"Que Pasa?\",\n",
        "                \"ROFL\": \"Rolling On The Floor Laughing\",\n",
        "                \"ROFLOL\": \"Rolling On The Floor Laughing Out Loud\",\n",
        "                \"ROTFLMAO\": \"Rolling On The Floor Laughing My Ass Off\",\n",
        "                \"SK8\": \"Skate\",\n",
        "                \"STATS\": \"Your sex and age\",\n",
        "                \"ASL\": \"Age, Sex, Location\",\n",
        "                \"THX\": \"Thank You\",\n",
        "                \"TTFN\": \"Bye bye\",\n",
        "                \"TTYL\": \"Talk To You Later\",\n",
        "                \"U\": \"You\",\n",
        "                \"U2\": \"You Too\",\n",
        "                \"U4E\": \"Yours For Ever\",\n",
        "                \"WB\": \"Welcome Back\",\n",
        "                \"WTF\": \"What The Fuck\",\n",
        "                \"WTG\": \"Way To Go!\",\n",
        "                \"WUF\": \"Where Are You From?\",\n",
        "                \"W8\": \"Wait\",\n",
        "                \"7K\": \"Sick Laugher\",\n",
        "                \"BBF\": \"Best friend forever\",\n",
        "                \"CUZ\": 'because',\n",
        "                \"BAE\": 'boyfriend',\n",
        "                \"LOV\": 'love',\n",
        "                \"THO\": 'though',\n",
        "                \"<3\": 'love',\n",
        "                'DM': 'direct message',\n",
        "                'DMING': 'direct messaging',\n",
        "                'DMED': 'direct messaged',\n",
        "                'PLZ': 'please',\n",
        "                'Y': 'why',\n",
        "                'L8': 'late',\n",
        "                'NIGGA': 'black man',\n",
        "                'NIGGAS': 'black men',\n",
        "                'DIS': \"disrespect\",\n",
        "                \"DAT\": 'that',\n",
        "                \"LUV\": \"love\",\n",
        "                \"SRRY\": \"sorry\",\n",
        "                \"TIL\": \"until\",\n",
        "                \"AWE\": \"awesome\",\n",
        "                \"OMG\": \"oh my god\"}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QkjXFzJoYS0O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def warmup_linear_schedule(optimizer, warmup_steps, t_total, last_epoch=-1):\n",
        "    def lr_lambda(step):\n",
        "        if step < warmup_steps:\n",
        "            return float(step) / float(max(1, warmup_steps))\n",
        "        return max(0.0, float(t_total - step) / float(max(1.0, t_total - warmup_steps)))\n",
        "\n",
        "    return LambdaLR(optimizer, lr_lambda, last_epoch=-1)\n",
        "\n",
        "class ELMoEmbedding(BaseEstimator, TransformerMixin):\n",
        "    def transform(self, X, y=None):\n",
        "        # Data preprocessing\n",
        "        X_clean = list(cont.expand_texts(X, precise=True))\n",
        "        X_clean = [tweet_clean(tweet) for tweet in X_clean]\n",
        "        sentences = [tweet.split() for tweet in X_clean]\n",
        "        # Model warm-up\n",
        "        _ = ee.embed_sentences(sentences[:100], batch_size=32)\n",
        "        # Get embeddings\n",
        "        pbar = tqdm(total=len(sentences))\n",
        "        ee_embeddings = []\n",
        "        for sentence, embedding in zip(sentences, ee.embed_sentences(sentences, batch_size=32)):\n",
        "            ee_embeddings.append(embedding[-1,:,:].mean(axis=0))\n",
        "            pbar.update(1)\n",
        "        pbar.close()\n",
        "        ee_embeddings = np.vstack(ee_embeddings)\n",
        "        return ee_embeddings\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "        return self\n",
        "\n",
        "\n",
        "class BERTEmbedding(BaseEstimator, TransformerMixin):\n",
        "    def transform(self, X, y=None):\n",
        "        # Data preprocessing\n",
        "        X_clean = list(cont.expand_texts(X, precise=True))\n",
        "        X_clean = [tweet_clean(tweet) for tweet in X_clean]\n",
        "        # Tokens to words ids and attention masks\n",
        "        input_ids = []\n",
        "        attention_masks = []\n",
        "        for sent in X_clean:\n",
        "            encoded_dict = tokenizer.encode_plus(sent, add_special_tokens=True, max_length=self.tokens_length, pad_to_max_length=True, return_attention_mask=True, return_tensors='pt')  \n",
        "            input_ids.append(encoded_dict['input_ids'])\n",
        "            attention_masks.append(encoded_dict['attention_mask'])\n",
        "        input_ids = torch.cat(input_ids, dim=0)\n",
        "        attention_masks = torch.cat(attention_masks, dim=0)\n",
        "        # Dataloader\n",
        "        batch_size = 32\n",
        "        dataset = TensorDataset(input_ids, attention_masks)\n",
        "        dataloader = DataLoader(dataset, sampler=SequentialSampler(dataset), batch_size=batch_size)\n",
        "        # Get embeddings\n",
        "        embedding = []\n",
        "        self.model.eval()\n",
        "        with torch.no_grad():\n",
        "          for batch in dataloader:\n",
        "                batch_ids = batch[0].to(device)\n",
        "                batch_mask = batch[1].to(device)       \n",
        "                _, batch_emb = self.model(batch_ids, token_type_ids=None, attention_mask=batch_mask)\n",
        "                embedding.append(batch_emb[-1][:,0,:].cpu().detach())\n",
        "                del _, batch_emb, batch_ids, batch_mask\n",
        "        # Deallocates model\n",
        "        if self.del_cnt > 0:\n",
        "            print('Delete')\n",
        "            del self.model\n",
        "            torch.cuda.empty_cache()\n",
        "        self.del_cnt += 1\n",
        "        bert_embeddings = torch.cat(embedding).numpy()\n",
        "        return torch.cat(embedding).cpu().detach().numpy()\n",
        "\n",
        "    def fit(self, X, y=None):\n",
        "        self.del_cnt = 0\n",
        "        dic = {'low': 0, 'medium': 1, 'high': 2}\n",
        "        y_class = [dic[class_] for class_ in y]\n",
        "        # Data preprocessing\n",
        "        X_clean = list(cont.expand_texts(X, precise=True))\n",
        "        X_clean = [tweet_clean(tweet) for tweet in X_clean]\n",
        "        # Sentence max length\n",
        "        max_length = 0\n",
        "        for sent in X_clean:\n",
        "            input_ids = tokenizer.encode(sent, add_special_tokens=True)\n",
        "            max_length = max(max_length, len(input_ids))\n",
        "        self.tokens_length = max_length + 10\n",
        "        # Tokens to words ids and attention masks\n",
        "        input_ids = []\n",
        "        attention_masks = []\n",
        "        for sent in X_clean:\n",
        "            encoded_dict = tokenizer.encode_plus(sent, add_special_tokens=True, max_length=self.tokens_length, pad_to_max_length=True, return_attention_mask=True, return_tensors='pt')  \n",
        "            input_ids.append(encoded_dict['input_ids'])\n",
        "            attention_masks.append(encoded_dict['attention_mask'])\n",
        "        input_ids = torch.cat(input_ids, dim=0)\n",
        "        attention_masks = torch.cat(attention_masks, dim=0)\n",
        "        labels = torch.LongTensor(y_class)\n",
        "        # Dataloader\n",
        "        batch_size = 32\n",
        "        dataset = TensorDataset(input_ids, attention_masks, labels)\n",
        "        dataloader = DataLoader(dataset, sampler=RandomSampler(dataset), batch_size=batch_size)\n",
        "        # BERT pre-trained model loading\n",
        "        model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=len(np.unique(y)), output_attentions=False, output_hidden_states=True)\n",
        "        model.cuda()\n",
        "        # Adam optimizer with weight decay\n",
        "        no_decay = ['bias', 'LayerNorm.weight'] \n",
        "        optimizer_grouped_parameters = [\n",
        "            {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n",
        "            'weight_decay_rate': 0.1},\n",
        "            {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n",
        "            'weight_decay_rate': 0.0}\n",
        "        ]\n",
        "        optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5, eps=1e-8)\n",
        "        # Learning rate decreasing schedule\n",
        "        epochs = 8\n",
        "        total_steps = len(dataloader) * epochs\n",
        "        scheduler = warmup_linear_schedule(optimizer, 0, total_steps)\n",
        "        #scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=total_steps)\n",
        "        # Training loop\n",
        "        start_time = time.time()\n",
        "        print(\"Training BERT embeddings...\")\n",
        "        for i in range(epochs):\n",
        "            train_accuracy = 0\n",
        "            train_loss = 0\n",
        "            # Train\n",
        "            model.train()\n",
        "            for step, batch in enumerate(dataloader):\n",
        "                batch_ids = batch[0].to(device)\n",
        "                batch_mask = batch[1].to(device)\n",
        "                batch_labels = batch[2].to(device)\n",
        "                optimizer.zero_grad()\n",
        "                loss, logits, _ = model(batch_ids, token_type_ids=None, attention_mask=batch_mask, labels=batch_labels)\n",
        "                train_loss += loss.item()\n",
        "                train_accuracy += accuracy_score(batch_labels.cpu().numpy(), logits.detach().cpu().numpy().argmax(axis=1))\n",
        "                loss.backward()\n",
        "                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "                optimizer.step()\n",
        "                scheduler.step()\n",
        "                del batch_ids, batch_mask, batch_labels, loss, logits, _\n",
        "            train_accuracy /= len(dataloader)\n",
        "            train_loss /= len(dataloader)\n",
        "            print(\"Epoch: {} | Acc: {} | Loss: {}\".format(i+1, train_accuracy, train_loss))\n",
        "        self.model = model\n",
        "        del scheduler, optimizer\n",
        "        return self"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-07T15:44:21.155528Z",
          "start_time": "2020-04-07T15:44:21.149545Z"
        },
        "id": "4EltyftEQ1rx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.ensemble import RandomForestClassifier, VotingClassifier\n",
        "\n",
        "def get_experiment_0_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('bow', CountVectorizer()),\n",
        "                                    ('chars_count', CharsCountTransformer())\n",
        "                                    ])), ('clf', MultinomialNB())])\n",
        "    \n",
        "def ELMo_pipeline():\n",
        "    return Pipeline([('features', ELMoEmbedding()), ('clf', MLPClassifier(hidden_layer_sizes=(300, 60)))])\n",
        "\n",
        "def ELMo_RF_pipeline():\n",
        "    return Pipeline([('features', ELMoEmbedding()), ('clf', RandomForestClassifier())])\n",
        "\n",
        "def BERT_pipeline():\n",
        "    return Pipeline([('features', BERTEmbedding()), ('clf', MLPClassifier(hidden_layer_sizes=(300, 60)))])\n",
        "\n",
        "def ELMo_MLP_pipeline():\n",
        "    return Pipeline([('features', ELMoEmbedding()), ('clf', MLPClassifier(hidden_layer_sizes=(300, 60)))])\n",
        "\n",
        "def RFs_pipeline():\n",
        "    return Pipeline([('features', BERTEmbedding()), ('clf', VotingClassifier(estimators=[('rf1', RandomForestClassifier()), ('rf2', RandomForestClassifier()), ('rf3', RandomForestClassifier()), ('rf4', RandomForestClassifier()), ('rf5', RandomForestClassifier()), ('rf6', RandomForestClassifier()), ('rf7', RandomForestClassifier()), ('rf8', RandomForestClassifier()), ('rf9', RandomForestClassifier()), ('rf10', RandomForestClassifier())], voting='soft'))])\n",
        "\n",
        "def RFs_bal_pipeline():\n",
        "    return Pipeline([('features', BERTEmbedding()), ('clf', VotingClassifier(estimators=[('rf1', RandomForestClassifier(class_weight='balanced')), ('rf2', RandomForestClassifier(class_weight='balanced')), ('rf3', RandomForestClassifier(class_weight='balanced')), ('rf4', RandomForestClassifier(class_weight='balanced')), ('rf5', RandomForestClassifier(class_weight='balanced')), ('rf6', RandomForestClassifier(class_weight='balanced')), ('rf7', RandomForestClassifier(class_weight='balanced')), ('rf8', RandomForestClassifier(class_weight='balanced')), ('rf9', RandomForestClassifier(class_weight='balanced')), ('rf10', RandomForestClassifier(class_weight='balanced'))], voting='soft'))])\n",
        "\n",
        "def MLPs_pipeline():\n",
        "    return Pipeline([('features', BERTEmbedding()), ('clf', VotingClassifier(estimators=[('rf1', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf2', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf3', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf4', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf5', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf6', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf7', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf8', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf9', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf10', MLPClassifier(hidden_layer_sizes=(400, 60)))], voting='soft'))])\n",
        "\n",
        "def RFs_MLPs_pipeline():\n",
        "    return Pipeline([('features', BERTEmbedding()), ('clf', VotingClassifier(estimators=[('rf1', RandomForestClassifier()), ('rf2', RandomForestClassifier()), ('rf3', RandomForestClassifier()), ('rf4', RandomForestClassifier()), ('rf5', RandomForestClassifier()), ('rf6', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf7', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf8', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf9', MLPClassifier(hidden_layer_sizes=(400, 60))), ('rf10', MLPClassifier(hidden_layer_sizes=(400, 60)))], voting='soft'))])\n",
        "\n",
        "def ELMo_RFs_pipeline():\n",
        "    return Pipeline([('features', ELMoEmbedding()), ('clf', VotingClassifier(estimators=[('rf1', RandomForestClassifier()), ('rf2', RandomForestClassifier()), ('rf3', RandomForestClassifier()), ('rf4', RandomForestClassifier()), ('rf5', RandomForestClassifier()), ('rf6', RandomForestClassifier()), ('rf7', RandomForestClassifier()), ('rf8', RandomForestClassifier()), ('rf9', RandomForestClassifier()), ('rf10', RandomForestClassifier())], voting='soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vKyGx-Ge6ZP1",
        "colab_type": "code",
        "outputId": "1d12f0c6-097d-4b61-dd0f-4bb03014add7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "2affab2bb1bc4f0ab18b19b06ecafb30",
            "19306f59bc914c2791ff99e7487fdeff",
            "f5addb3d576243fc8d83844b3872a546",
            "a6390a4808b340e9bb955ddd1fe26d1a",
            "14f51065f0944d26a18ed2ffc12d5f96",
            "91d76251add3430b952ab1186e8876c4",
            "ef2a8771168648479fa1c308046b6040",
            "9e2ff4cfac2646ada0c0c98b774ea26c",
            "48edb44ab64a4b9aa42d31de7187f802",
            "3363500ff87748ed9d498acff0e19846",
            "4503b2b8cef6488c8729667dbc943fcf",
            "4d2716aa32944b949982b8a26ecb6029",
            "20ccfce4f0c54cbc87e51012bcb8524a",
            "ece3796c310243e8ae622b8fd52f3c1a",
            "e9d66bd612bc4880b1ef0e8c67ee1194",
            "1de5962bed8b41139ad75b909dc9d6d4",
            "879988b407a9447584bbbd6bb7f725e6",
            "88c70189d9404f23962995ef2c6b6da9",
            "6bc6b43d80d54227af173382ff89fe74",
            "24b02d4e08ff402e8964a3ed35f65943",
            "b6f96f53e3844d0ebee8d0c8485ad4df",
            "b6f93012fe024047966c6db0bb8a7536",
            "78c62cc04fc0403e876e3117f63ba199",
            "3369f95e02b94466ab0191ff4f7f491b",
            "a5bb987053e04fd8a31cd05c328eed5c",
            "2f94713df08a40988f6c99a3293fc475",
            "792042edb6f94e27a66acf87c16f92e9",
            "db8d2438187340b5a7e369118c0877c9",
            "ba923abf524d4a1999a890340281b1a6",
            "c496ec4601ea4a78add195f7471e5187",
            "2bd4d395647345e5bc1bb572eb3d8195",
            "1ac1e87d5d1a4dfd83769948b291324a",
            "aa1eee7900484ae5bfea925a5ed76249",
            "df8fd1add3a54ed8bd6f58f90657aa56",
            "4e138c81ce684d6c9ffc2565a3fe8949",
            "e1b99e7d9cca44b1a1175d44f08a5d90",
            "7f551412c693455295381cb271e06de7",
            "05af46e7c8f54759af11c04d333ba839",
            "25c26d6b4ae94fd193f923b3627d3c0e",
            "3e65840d8a3f490cb7205a66e8e47b55",
            "fcad0ad880754c0eacf418abdbb6c887",
            "6256a0614ee642f18fc22aa34d5178f6",
            "8e630d49da67482599a5a52da0a167f7",
            "9f2273218a74479ebb3e6bca83e15d7e",
            "ff78ce7d01cb4fc887dfd0b7e4fce882",
            "277e0ecbe82c4c6a8cb486cb0c6790f6",
            "8d80d827a93d487184288a0f7f0c844c",
            "81dd3288cb4b4192a17132dcc9ca13bb",
            "206b267df071405487e17fb827552882",
            "47ad5d427ba64052976436880c156a80",
            "87057898c3b144cd9c86f361bf00ebae",
            "3b7724c4a58e4b679995d70150fd70ec",
            "643d0524ced14525845e32d3f872ae90",
            "0e6c3dee475c47919c8a363a46e3a91b",
            "aa69d81d67b145808e6dc0daa7868fe7",
            "2da4fdb1fc51412a9f513ccd0c59df67",
            "3f10fecdf3f244139923a148da60f0f0",
            "73bf863b8843449b94966d7144d12356",
            "1d897157019247c2bd51bca83349171e",
            "ed34325a0d2149a3a63e61e14f19c9db",
            "d72e7cf7676d4a7eb5aba532e94359d5",
            "e5312ae4086b43df8bf04e82864494a2",
            "42440ba51d1e4929a1e76f32f4846443",
            "6ae1f473f78a420e8b01956f7c76e5b9"
          ]
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    #pipeline = get_experiment_0_pipeline()\n",
        "    #pipeline = ELMo_pipeline()\n",
        "    #pipeline = BERT_pipeline()\n",
        "    #pipeline = RFs_pipeline()\n",
        "    #pipeline = RFs_bal_pipeline()\n",
        "    #pipeline = MLPs_pipeline()\n",
        "    #pipeline = RFs_MLPs_pipeline()\n",
        "    pipeline = ELMo_RFs_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "    #classifier, learned_labels, scores = run_test(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2affab2bb1bc4f0ab18b19b06ecafb30",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=630.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "48edb44ab64a4b9aa42d31de7187f802",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=311.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[  2  51   0]\n",
            " [  4 197   3]\n",
            " [  0  49   5]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.33      0.04      0.07        53\n",
            "      medium       0.66      0.97      0.79       204\n",
            "        high       0.62      0.09      0.16        54\n",
            "\n",
            "    accuracy                           0.66       311\n",
            "   macro avg       0.54      0.37      0.34       311\n",
            "weighted avg       0.60      0.66      0.56       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.675\tKappa: 0.06\tAccuracy: 0.656\n",
            "------------------------------------------------------\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "879988b407a9447584bbbd6bb7f725e6",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=842.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a5bb987053e04fd8a31cd05c328eed5c",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=415.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 14  81   0]\n",
            " [  4 223   4]\n",
            " [  1  71  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.74      0.15      0.25        95\n",
            "      medium       0.59      0.97      0.74       231\n",
            "        high       0.81      0.19      0.31        89\n",
            "\n",
            "    accuracy                           0.61       415\n",
            "   macro avg       0.71      0.43      0.43       415\n",
            "weighted avg       0.67      0.61      0.53       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.698\tKappa: 0.184\tAccuracy: 0.612\n",
            "------------------------------------------------------\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "aa1eee7900484ae5bfea925a5ed76249",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=604.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fcad0ad880754c0eacf418abdbb6c887",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=298.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[  5  67   0]\n",
            " [  1 157   3]\n",
            " [  0  43  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.83      0.07      0.13        72\n",
            "      medium       0.59      0.98      0.73       161\n",
            "        high       0.88      0.34      0.49        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.77      0.46      0.45       298\n",
            "weighted avg       0.71      0.62      0.53       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.748\tKappa: 0.224\tAccuracy: 0.617\n",
            "------------------------------------------------------\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "206b267df071405487e17fb827552882",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=576.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3f10fecdf3f244139923a148da60f0f0",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=284.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 15  54   0]\n",
            " [  6 141   3]\n",
            " [  0  57   8]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.71      0.22      0.33        69\n",
            "      medium       0.56      0.94      0.70       150\n",
            "        high       0.73      0.12      0.21        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.67      0.43      0.42       284\n",
            "weighted avg       0.64      0.58      0.50       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.678\tKappa: 0.162\tAccuracy: 0.577\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.7\t Average Kappa: 0.158\t Average Accuracy: 0.615\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zqjxMme5wjAn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_47_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(300, 60, ), random_state=1138))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gvqKmnD8xTsZ",
        "colab_type": "code",
        "outputId": "09125603-dc8b-4c2d-f974-70549c5b03a1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_47_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.6221590909090909 | Loss: 0.9288951635360718\n",
            "Epoch: 2 | Acc: 0.6545454545454545 | Loss: 0.8779470354318619\n",
            "Epoch: 3 | Acc: 0.6676136363636364 | Loss: 0.8163020640611649\n",
            "Epoch: 4 | Acc: 0.7105113636363637 | Loss: 0.7164434015750885\n",
            "Epoch: 5 | Acc: 0.7829545454545455 | Loss: 0.5826232314109803\n",
            "Epoch: 6 | Acc: 0.8548295454545455 | Loss: 0.48631281554698946\n",
            "Epoch: 7 | Acc: 0.8985795454545455 | Loss: 0.42254353910684583\n",
            "Epoch: 8 | Acc: 0.9181818181818182 | Loss: 0.38362365365028384\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 17  36   0]\n",
            " [ 17 172  15]\n",
            " [  1  32  21]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.49      0.32      0.39        53\n",
            "      medium       0.72      0.84      0.77       204\n",
            "        high       0.58      0.39      0.47        54\n",
            "\n",
            "    accuracy                           0.68       311\n",
            "   macro avg       0.60      0.52      0.54       311\n",
            "weighted avg       0.65      0.68      0.66       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.66\tKappa: 0.285\tAccuracy: 0.675\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5203703703703704 | Loss: 1.006606561166269\n",
            "Epoch: 2 | Acc: 0.5604166666666667 | Loss: 0.9329814226539047\n",
            "Epoch: 3 | Acc: 0.6344907407407407 | Loss: 0.7919419761057254\n",
            "Epoch: 4 | Acc: 0.7400462962962963 | Loss: 0.6229959472461983\n",
            "Epoch: 5 | Acc: 0.8192129629629629 | Loss: 0.48933526542451644\n",
            "Epoch: 6 | Acc: 0.8768518518518519 | Loss: 0.4023058436535023\n",
            "Epoch: 7 | Acc: 0.9060185185185184 | Loss: 0.341710662400281\n",
            "Epoch: 8 | Acc: 0.9305555555555556 | Loss: 0.28673953425001214\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 50  41   4]\n",
            " [ 23 194  14]\n",
            " [  0  44  45]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.68      0.53      0.60        95\n",
            "      medium       0.70      0.84      0.76       231\n",
            "        high       0.71      0.51      0.59        89\n",
            "\n",
            "    accuracy                           0.70       415\n",
            "   macro avg       0.70      0.62      0.65       415\n",
            "weighted avg       0.70      0.70      0.69       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.816\tKappa: 0.451\tAccuracy: 0.696\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5028195488721805 | Loss: 1.0176589708579213\n",
            "Epoch: 2 | Acc: 0.6043233082706767 | Loss: 0.8779300514020418\n",
            "Epoch: 3 | Acc: 0.700187969924812 | Loss: 0.6808433909165231\n",
            "Epoch: 4 | Acc: 0.7866541353383458 | Loss: 0.5512261908305319\n",
            "Epoch: 5 | Acc: 0.849624060150376 | Loss: 0.4675827951807725\n",
            "Epoch: 6 | Acc: 0.8898026315789473 | Loss: 0.4037865619910391\n",
            "Epoch: 7 | Acc: 0.9292763157894737 | Loss: 0.3561350813037471\n",
            "Epoch: 8 | Acc: 0.9243421052631579 | Loss: 0.33135581565530675\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 30  42   0]\n",
            " [ 20 117  24]\n",
            " [  0  31  34]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.42      0.49        72\n",
            "      medium       0.62      0.73      0.67       161\n",
            "        high       0.59      0.52      0.55        65\n",
            "\n",
            "    accuracy                           0.61       298\n",
            "   macro avg       0.60      0.56      0.57       298\n",
            "weighted avg       0.61      0.61      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.72\tKappa: 0.314\tAccuracy: 0.607\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5208333333333334 | Loss: 1.021331661277347\n",
            "Epoch: 2 | Acc: 0.5625 | Loss: 0.9058141178554959\n",
            "Epoch: 3 | Acc: 0.6805555555555556 | Loss: 0.733488185538186\n",
            "Epoch: 4 | Acc: 0.7916666666666666 | Loss: 0.569810774591234\n",
            "Epoch: 5 | Acc: 0.8836805555555556 | Loss: 0.46502627929051715\n",
            "Epoch: 6 | Acc: 0.8732638888888888 | Loss: 0.4199197010861503\n",
            "Epoch: 7 | Acc: 0.8993055555555556 | Loss: 0.3614832858244578\n",
            "Epoch: 8 | Acc: 0.9375 | Loss: 0.3171757956345876\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 22  45   2]\n",
            " [ 16 117  17]\n",
            " [  0  39  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.32      0.41        69\n",
            "      medium       0.58      0.78      0.67       150\n",
            "        high       0.58      0.40      0.47        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.58      0.50      0.52       284\n",
            "weighted avg       0.58      0.58      0.56       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.723\tKappa: 0.248\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.73\t Average Kappa: 0.325\t Average Accuracy: 0.64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zQYYbIKpY3eT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_48_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VAPES607gnj_",
        "colab_type": "code",
        "outputId": "ac33e264-afd6-4ee4-ecd0-96684894c2aa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_48_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5823863636363636 | Loss: 0.9339311063289643\n",
            "Epoch: 2 | Acc: 0.653125 | Loss: 0.8301039963960648\n",
            "Epoch: 3 | Acc: 0.6934659090909091 | Loss: 0.7110916107892991\n",
            "Epoch: 4 | Acc: 0.766903409090909 | Loss: 0.5988997370004654\n",
            "Epoch: 5 | Acc: 0.8339488636363637 | Loss: 0.48446888774633406\n",
            "Epoch: 6 | Acc: 0.8890625 | Loss: 0.41019905880093577\n",
            "Epoch: 7 | Acc: 0.9171875 | Loss: 0.3427965804934502\n",
            "Epoch: 8 | Acc: 0.9336647727272727 | Loss: 0.3176754005253315\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 15  38   0]\n",
            " [ 17 174  13]\n",
            " [  0  37  17]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.47      0.28      0.35        53\n",
            "      medium       0.70      0.85      0.77       204\n",
            "        high       0.57      0.31      0.40        54\n",
            "\n",
            "    accuracy                           0.66       311\n",
            "   macro avg       0.58      0.48      0.51       311\n",
            "weighted avg       0.64      0.66      0.63       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.645\tKappa: 0.234\tAccuracy: 0.662\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5643518518518519 | Loss: 0.980301731162601\n",
            "Epoch: 2 | Acc: 0.5886574074074075 | Loss: 0.9015312548036929\n",
            "Epoch: 3 | Acc: 0.6692129629629631 | Loss: 0.7462813169867905\n",
            "Epoch: 4 | Acc: 0.7844907407407407 | Loss: 0.5986217138943849\n",
            "Epoch: 5 | Acc: 0.8641203703703704 | Loss: 0.465438617600335\n",
            "Epoch: 6 | Acc: 0.9046296296296297 | Loss: 0.36556249305053995\n",
            "Epoch: 7 | Acc: 0.9236111111111112 | Loss: 0.2970698771101457\n",
            "Epoch: 8 | Acc: 0.9381944444444444 | Loss: 0.27267896080458603\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 45  47   3]\n",
            " [ 23 185  23]\n",
            " [  1  44  44]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.47      0.55        95\n",
            "      medium       0.67      0.80      0.73       231\n",
            "        high       0.63      0.49      0.55        89\n",
            "\n",
            "    accuracy                           0.66       415\n",
            "   macro avg       0.65      0.59      0.61       415\n",
            "weighted avg       0.66      0.66      0.65       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.795\tKappa: 0.388\tAccuracy: 0.66\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.537359022556391 | Loss: 1.0010300655114024\n",
            "Epoch: 2 | Acc: 0.6205357142857143 | Loss: 0.8571230298594424\n",
            "Epoch: 3 | Acc: 0.7589285714285715 | Loss: 0.6822527741131029\n",
            "Epoch: 4 | Acc: 0.8583176691729324 | Loss: 0.5154513120651245\n",
            "Epoch: 5 | Acc: 0.8893327067669172 | Loss: 0.43228664053113836\n",
            "Epoch: 6 | Acc: 0.9168233082706767 | Loss: 0.3468153570827685\n",
            "Epoch: 7 | Acc: 0.9572368421052632 | Loss: 0.28666923626473073\n",
            "Epoch: 8 | Acc: 0.9720394736842105 | Loss: 0.2505142649537639\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 31  39   2]\n",
            " [ 26 113  22]\n",
            " [  1  29  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.43      0.48        72\n",
            "      medium       0.62      0.70      0.66       161\n",
            "        high       0.59      0.54      0.56        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.58      0.56      0.57       298\n",
            "weighted avg       0.60      0.60      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.715\tKappa: 0.313\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5243055555555556 | Loss: 1.01788819498486\n",
            "Epoch: 2 | Acc: 0.5625 | Loss: 0.965515649980969\n",
            "Epoch: 3 | Acc: 0.6996527777777778 | Loss: 0.8296584950553046\n",
            "Epoch: 4 | Acc: 0.7708333333333334 | Loss: 0.6841663850678338\n",
            "Epoch: 5 | Acc: 0.8576388888888888 | Loss: 0.5359085847934087\n",
            "Epoch: 6 | Acc: 0.9079861111111112 | Loss: 0.4266228940751817\n",
            "Epoch: 7 | Acc: 0.9236111111111112 | Loss: 0.3642186083727413\n",
            "Epoch: 8 | Acc: 0.9340277777777778 | Loss: 0.32945677803622353\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[35 32  2]\n",
            " [27 93 30]\n",
            " [ 7 23 35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.51      0.51        69\n",
            "      medium       0.63      0.62      0.62       150\n",
            "        high       0.52      0.54      0.53        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.55      0.56      0.55       284\n",
            "weighted avg       0.57      0.57      0.57       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.718\tKappa: 0.304\tAccuracy: 0.574\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.718\t Average Kappa: 0.31\t Average Accuracy: 0.624\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ljKMOF8d8fod",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_49_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(300, 60, ), random_state=1138)),\n",
        "                                                                   ('MLP2', MLP(hidden_layer_sizes=(300, 60, )))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNQ0XqEq8rSJ",
        "colab_type": "code",
        "outputId": "ff71a0b5-155e-4135-f90f-f82ccb61294c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_49_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.6217329545454546 | Loss: 0.9155984580516815\n",
            "Epoch: 2 | Acc: 0.653125 | Loss: 0.850255885720253\n",
            "Epoch: 3 | Acc: 0.6832386363636364 | Loss: 0.7137973606586456\n",
            "Epoch: 4 | Acc: 0.7948863636363637 | Loss: 0.5450101479887962\n",
            "Epoch: 5 | Acc: 0.8711647727272727 | Loss: 0.3964427947998047\n",
            "Epoch: 6 | Acc: 0.9042613636363637 | Loss: 0.3216349795460701\n",
            "Epoch: 7 | Acc: 0.9477272727272726 | Loss: 0.2506950058043003\n",
            "Epoch: 8 | Acc: 0.9609375 | Loss: 0.2245512567460537\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 19  34   0]\n",
            " [ 12 180  12]\n",
            " [  0  29  25]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.61      0.36      0.45        53\n",
            "      medium       0.74      0.88      0.81       204\n",
            "        high       0.68      0.46      0.55        54\n",
            "\n",
            "    accuracy                           0.72       311\n",
            "   macro avg       0.68      0.57      0.60       311\n",
            "weighted avg       0.71      0.72      0.70       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.724\tKappa: 0.378\tAccuracy: 0.72\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5372685185185185 | Loss: 1.0043545343257763\n",
            "Epoch: 2 | Acc: 0.5601851851851852 | Loss: 0.9364929419976694\n",
            "Epoch: 3 | Acc: 0.6844907407407407 | Loss: 0.7802161243226793\n",
            "Epoch: 4 | Acc: 0.779861111111111 | Loss: 0.615942464934455\n",
            "Epoch: 5 | Acc: 0.8553240740740741 | Loss: 0.4783562476988192\n",
            "Epoch: 6 | Acc: 0.9016203703703703 | Loss: 0.3779384162690904\n",
            "Epoch: 7 | Acc: 0.9152777777777777 | Loss: 0.3243272950251897\n",
            "Epoch: 8 | Acc: 0.9282407407407407 | Loss: 0.2957051578495238\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 57  33   5]\n",
            " [ 26 178  27]\n",
            " [  1  50  38]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.68      0.60      0.64        95\n",
            "      medium       0.68      0.77      0.72       231\n",
            "        high       0.54      0.43      0.48        89\n",
            "\n",
            "    accuracy                           0.66       415\n",
            "   macro avg       0.63      0.60      0.61       415\n",
            "weighted avg       0.65      0.66      0.65       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.764\tKappa: 0.397\tAccuracy: 0.658\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4866071428571428 | Loss: 1.0357260515815334\n",
            "Epoch: 2 | Acc: 0.5493421052631579 | Loss: 0.939696402926194\n",
            "Epoch: 3 | Acc: 0.6207706766917294 | Loss: 0.7874051112877695\n",
            "Epoch: 4 | Acc: 0.7140507518796992 | Loss: 0.6504641332124409\n",
            "Epoch: 5 | Acc: 0.7915883458646616 | Loss: 0.548758654217971\n",
            "Epoch: 6 | Acc: 0.8479793233082706 | Loss: 0.46630406693408366\n",
            "Epoch: 7 | Acc: 0.894266917293233 | Loss: 0.40453524652280304\n",
            "Epoch: 8 | Acc: 0.906015037593985 | Loss: 0.39157473570422124\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 24  47   1]\n",
            " [ 17 126  18]\n",
            " [  1  30  34]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.33      0.42        72\n",
            "      medium       0.62      0.78      0.69       161\n",
            "        high       0.64      0.52      0.58        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.61      0.55      0.56       298\n",
            "weighted avg       0.61      0.62      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.707\tKappa: 0.316\tAccuracy: 0.617\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4878472222222222 | Loss: 1.0327217015955183\n",
            "Epoch: 2 | Acc: 0.5295138888888888 | Loss: 0.9376172423362732\n",
            "Epoch: 3 | Acc: 0.5885416666666666 | Loss: 0.8304215967655182\n",
            "Epoch: 4 | Acc: 0.6857638888888888 | Loss: 0.7099942730532752\n",
            "Epoch: 5 | Acc: 0.7586805555555556 | Loss: 0.6018035146925185\n",
            "Epoch: 6 | Acc: 0.8368055555555556 | Loss: 0.511563691827986\n",
            "Epoch: 7 | Acc: 0.8854166666666666 | Loss: 0.4448732915851805\n",
            "Epoch: 8 | Acc: 0.8993055555555556 | Loss: 0.4221542775630951\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 30  38   1]\n",
            " [ 23 113  14]\n",
            " [  1  37  27]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.43      0.49        69\n",
            "      medium       0.60      0.75      0.67       150\n",
            "        high       0.64      0.42      0.50        65\n",
            "\n",
            "    accuracy                           0.60       284\n",
            "   macro avg       0.60      0.53      0.55       284\n",
            "weighted avg       0.60      0.60      0.59       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.725\tKappa: 0.296\tAccuracy: 0.599\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.73\t Average Kappa: 0.347\t Average Accuracy: 0.649\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "crkcNR9wAMgM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_50_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138)),\n",
        "                                                                   ('MLP2', MLP(hidden_layer_sizes=(2000, 500, 100, )))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mTiqwEJmAM1A",
        "colab_type": "code",
        "outputId": "08796a5b-bc77-4934-c97e-09a2ffcdc1da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_50_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.6200284090909091 | Loss: 0.926224434375763\n",
            "Epoch: 2 | Acc: 0.6502840909090909 | Loss: 0.8551601886749267\n",
            "Epoch: 3 | Acc: 0.6769886363636364 | Loss: 0.750262662768364\n",
            "Epoch: 4 | Acc: 0.7505681818181819 | Loss: 0.5883377462625503\n",
            "Epoch: 5 | Acc: 0.8447443181818182 | Loss: 0.45291932821273806\n",
            "Epoch: 6 | Acc: 0.8900568181818181 | Loss: 0.35008883774280547\n",
            "Epoch: 7 | Acc: 0.9119318181818181 | Loss: 0.2958687856793404\n",
            "Epoch: 8 | Acc: 0.9515625 | Loss: 0.24980142936110497\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 15  38   0]\n",
            " [ 15 183   6]\n",
            " [  0  35  19]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.28      0.36        53\n",
            "      medium       0.71      0.90      0.80       204\n",
            "        high       0.76      0.35      0.48        54\n",
            "\n",
            "    accuracy                           0.70       311\n",
            "   macro avg       0.66      0.51      0.55       311\n",
            "weighted avg       0.69      0.70      0.67       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.719\tKappa: 0.297\tAccuracy: 0.698\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5435185185185185 | Loss: 1.0095640067700986\n",
            "Epoch: 2 | Acc: 0.5613425925925926 | Loss: 0.9301361772749159\n",
            "Epoch: 3 | Acc: 0.6314814814814815 | Loss: 0.7758867696479514\n",
            "Epoch: 4 | Acc: 0.7356481481481482 | Loss: 0.6044850868207438\n",
            "Epoch: 5 | Acc: 0.8106481481481481 | Loss: 0.48212363653712803\n",
            "Epoch: 6 | Acc: 0.8643518518518518 | Loss: 0.39380693766805863\n",
            "Epoch: 7 | Acc: 0.9097222222222222 | Loss: 0.3197075719082797\n",
            "Epoch: 8 | Acc: 0.9317129629629629 | Loss: 0.277546297620844\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 48  44   3]\n",
            " [ 18 197  16]\n",
            " [  1  46  42]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.72      0.51      0.59        95\n",
            "      medium       0.69      0.85      0.76       231\n",
            "        high       0.69      0.47      0.56        89\n",
            "\n",
            "    accuracy                           0.69       415\n",
            "   macro avg       0.70      0.61      0.64       415\n",
            "weighted avg       0.69      0.69      0.68       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.808\tKappa: 0.436\tAccuracy: 0.692\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5364191729323309 | Loss: 0.99473826508773\n",
            "Epoch: 2 | Acc: 0.5634398496240601 | Loss: 0.8660137935688621\n",
            "Epoch: 3 | Acc: 0.6764567669172932 | Loss: 0.7161761930114344\n",
            "Epoch: 4 | Acc: 0.8256578947368421 | Loss: 0.566065030662637\n",
            "Epoch: 5 | Acc: 0.8909774436090224 | Loss: 0.43545074525632355\n",
            "Epoch: 6 | Acc: 0.9407894736842105 | Loss: 0.33612620203118576\n",
            "Epoch: 7 | Acc: 0.9551221804511277 | Loss: 0.2868896550253818\n",
            "Epoch: 8 | Acc: 0.9619360902255639 | Loss: 0.26161491870880127\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 30  42   0]\n",
            " [ 16 131  14]\n",
            " [  1  30  34]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.64      0.42      0.50        72\n",
            "      medium       0.65      0.81      0.72       161\n",
            "        high       0.71      0.52      0.60        65\n",
            "\n",
            "    accuracy                           0.65       298\n",
            "   macro avg       0.66      0.58      0.61       298\n",
            "weighted avg       0.66      0.65      0.64       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.747\tKappa: 0.381\tAccuracy: 0.654\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4913194444444444 | Loss: 1.0487636493311987\n",
            "Epoch: 2 | Acc: 0.5329861111111112 | Loss: 0.9752188722292582\n",
            "Epoch: 3 | Acc: 0.6024305555555556 | Loss: 0.8480081756909689\n",
            "Epoch: 4 | Acc: 0.671875 | Loss: 0.7434308562013838\n",
            "Epoch: 5 | Acc: 0.7291666666666666 | Loss: 0.6405981944666969\n",
            "Epoch: 6 | Acc: 0.7986111111111112 | Loss: 0.5445848670270708\n",
            "Epoch: 7 | Acc: 0.8402777777777778 | Loss: 0.5068900154696571\n",
            "Epoch: 8 | Acc: 0.8611111111111112 | Loss: 0.4616691834396786\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 22  46   1]\n",
            " [ 16 118  16]\n",
            " [  0  32  33]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.32      0.41        69\n",
            "      medium       0.60      0.79      0.68       150\n",
            "        high       0.66      0.51      0.57        65\n",
            "\n",
            "    accuracy                           0.61       284\n",
            "   macro avg       0.61      0.54      0.56       284\n",
            "weighted avg       0.61      0.61      0.59       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.72\tKappa: 0.305\tAccuracy: 0.609\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.748\t Average Kappa: 0.355\t Average Accuracy: 0.663\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o19RJWNFDQ8x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_51_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC4', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC5', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC6', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC7', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC8', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC9', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC10', RFC(n_estimators=200, class_weight = 'balanced'))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YCEDdrj5DbsA",
        "colab_type": "code",
        "outputId": "9e235687-a1ae-42a1-b233-2f992617d416",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_51_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of add_ is deprecated:\n",
            "\tadd_(Number alpha, Tensor other)\n",
            "Consider using one of the following signatures instead:\n",
            "\tadd_(Tensor other, *, Number alpha)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1 | Acc: 0.6053977272727272 | Loss: 0.9388154834508896\n",
            "Epoch: 2 | Acc: 0.653125 | Loss: 0.8359764754772187\n",
            "Epoch: 3 | Acc: 0.6606534090909091 | Loss: 0.7437231987714767\n",
            "Epoch: 4 | Acc: 0.7143465909090909 | Loss: 0.6119107246398926\n",
            "Epoch: 5 | Acc: 0.7822443181818182 | Loss: 0.5072208657860756\n",
            "Epoch: 6 | Acc: 0.8555397727272727 | Loss: 0.4169035702943802\n",
            "Epoch: 7 | Acc: 0.878125 | Loss: 0.35435411185026167\n",
            "Epoch: 8 | Acc: 0.9329545454545455 | Loss: 0.3092748992145061\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 19  33   1]\n",
            " [ 15 180   9]\n",
            " [  0  30  24]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.56      0.36      0.44        53\n",
            "      medium       0.74      0.88      0.81       204\n",
            "        high       0.71      0.44      0.55        54\n",
            "\n",
            "    accuracy                           0.72       311\n",
            "   macro avg       0.67      0.56      0.60       311\n",
            "weighted avg       0.70      0.72      0.70       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.735\tKappa: 0.371\tAccuracy: 0.717\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4863425925925926 | Loss: 1.037010927995046\n",
            "Epoch: 2 | Acc: 0.5523148148148148 | Loss: 0.9454047878583273\n",
            "Epoch: 3 | Acc: 0.63125 | Loss: 0.8290931030556008\n",
            "Epoch: 4 | Acc: 0.7162037037037037 | Loss: 0.6828077810781973\n",
            "Epoch: 5 | Acc: 0.7870370370370371 | Loss: 0.5685276080060888\n",
            "Epoch: 6 | Acc: 0.8233796296296296 | Loss: 0.4872453014055888\n",
            "Epoch: 7 | Acc: 0.8819444444444444 | Loss: 0.41124285923110115\n",
            "Epoch: 8 | Acc: 0.8898148148148147 | Loss: 0.3868872677838361\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 55  38   2]\n",
            " [ 26 196   9]\n",
            " [  2  49  38]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.66      0.58      0.62        95\n",
            "      medium       0.69      0.85      0.76       231\n",
            "        high       0.78      0.43      0.55        89\n",
            "\n",
            "    accuracy                           0.70       415\n",
            "   macro avg       0.71      0.62      0.64       415\n",
            "weighted avg       0.70      0.70      0.68       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.803\tKappa: 0.447\tAccuracy: 0.696\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.49107142857142855 | Loss: 1.027017000474428\n",
            "Epoch: 2 | Acc: 0.630874060150376 | Loss: 0.8163714377503646\n",
            "Epoch: 3 | Acc: 0.7241541353383458 | Loss: 0.6896733327915794\n",
            "Epoch: 4 | Acc: 0.7965225563909775 | Loss: 0.5731881543209678\n",
            "Epoch: 5 | Acc: 0.8721804511278196 | Loss: 0.4741551703528354\n",
            "Epoch: 6 | Acc: 0.893796992481203 | Loss: 0.4118334839218541\n",
            "Epoch: 7 | Acc: 0.918468045112782 | Loss: 0.3696697934677726\n",
            "Epoch: 8 | Acc: 0.9269266917293233 | Loss: 0.3514712303876877\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 34  37   1]\n",
            " [ 25 126  10]\n",
            " [  1  32  32]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.47      0.52        72\n",
            "      medium       0.65      0.78      0.71       161\n",
            "        high       0.74      0.49      0.59        65\n",
            "\n",
            "    accuracy                           0.64       298\n",
            "   macro avg       0.65      0.58      0.61       298\n",
            "weighted avg       0.65      0.64      0.64       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.732\tKappa: 0.372\tAccuracy: 0.644\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4565972222222222 | Loss: 1.0379536648591359\n",
            "Epoch: 2 | Acc: 0.5555555555555556 | Loss: 0.9193842874632941\n",
            "Epoch: 3 | Acc: 0.6458333333333334 | Loss: 0.8114200863573287\n",
            "Epoch: 4 | Acc: 0.7361111111111112 | Loss: 0.7063398791684045\n",
            "Epoch: 5 | Acc: 0.8142361111111112 | Loss: 0.5943500283691618\n",
            "Epoch: 6 | Acc: 0.8715277777777778 | Loss: 0.5069300747579999\n",
            "Epoch: 7 | Acc: 0.890625 | Loss: 0.444407108757231\n",
            "Epoch: 8 | Acc: 0.9149305555555556 | Loss: 0.41310711867279476\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 26  42   1]\n",
            " [ 19 113  18]\n",
            " [  1  38  26]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.38      0.45        69\n",
            "      medium       0.59      0.75      0.66       150\n",
            "        high       0.58      0.40      0.47        65\n",
            "\n",
            "    accuracy                           0.58       284\n",
            "   macro avg       0.58      0.51      0.53       284\n",
            "weighted avg       0.58      0.58      0.57       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.739\tKappa: 0.259\tAccuracy: 0.581\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.752\t Average Kappa: 0.362\t Average Accuracy: 0.659\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pA7xNgnlYneW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_52_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('MLP1', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP2', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP3', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP4', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP5', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP6', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP7', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP8', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP9', MLP(hidden_layer_sizes=(2000, 500, 100, ))),\n",
        "                                                                   ('MLP10', MLP(hidden_layer_sizes=(2000, 500, 100, )))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IZfVy7CPZXj4",
        "colab_type": "code",
        "outputId": "416ff888-40cc-4f4c-d649-e4e6e15286e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_52_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of add_ is deprecated:\n",
            "\tadd_(Number alpha, Tensor other)\n",
            "Consider using one of the following signatures instead:\n",
            "\tadd_(Tensor other, *, Number alpha)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1 | Acc: 0.5180397727272728 | Loss: 1.006325927376747\n",
            "Epoch: 2 | Acc: 0.6552556818181818 | Loss: 0.8074824184179306\n",
            "Epoch: 3 | Acc: 0.7083806818181818 | Loss: 0.6563403069972992\n",
            "Epoch: 4 | Acc: 0.8032670454545455 | Loss: 0.5110551357269287\n",
            "Epoch: 5 | Acc: 0.8680397727272726 | Loss: 0.3982912331819534\n",
            "Epoch: 6 | Acc: 0.9211647727272727 | Loss: 0.314421459287405\n",
            "Epoch: 7 | Acc: 0.9423295454545455 | Loss: 0.2645797520875931\n",
            "Epoch: 8 | Acc: 0.9546875 | Loss: 0.24377567768096925\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 18  34   1]\n",
            " [ 15 176  13]\n",
            " [  1  23  30]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.34      0.41        53\n",
            "      medium       0.76      0.86      0.81       204\n",
            "        high       0.68      0.56      0.61        54\n",
            "\n",
            "    accuracy                           0.72       311\n",
            "   macro avg       0.66      0.59      0.61       311\n",
            "weighted avg       0.70      0.72      0.71       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.702\tKappa: 0.399\tAccuracy: 0.72\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5428240740740741 | Loss: 0.999534355269538\n",
            "Epoch: 2 | Acc: 0.5986111111111112 | Loss: 0.886786694879885\n",
            "Epoch: 3 | Acc: 0.6819444444444445 | Loss: 0.7539287121207626\n",
            "Epoch: 4 | Acc: 0.7807870370370371 | Loss: 0.5850852242222538\n",
            "Epoch: 5 | Acc: 0.8476851851851852 | Loss: 0.46059737713248644\n",
            "Epoch: 6 | Acc: 0.8981481481481481 | Loss: 0.3522524701224433\n",
            "Epoch: 7 | Acc: 0.9129629629629629 | Loss: 0.2980159443837625\n",
            "Epoch: 8 | Acc: 0.9236111111111112 | Loss: 0.2675836566421721\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 50  40   5]\n",
            " [ 31 189  11]\n",
            " [  2  55  32]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.53      0.56        95\n",
            "      medium       0.67      0.82      0.73       231\n",
            "        high       0.67      0.36      0.47        89\n",
            "\n",
            "    accuracy                           0.65       415\n",
            "   macro avg       0.64      0.57      0.59       415\n",
            "weighted avg       0.65      0.65      0.64       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.729\tKappa: 0.367\tAccuracy: 0.653\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5162124060150376 | Loss: 1.0219568767045673\n",
            "Epoch: 2 | Acc: 0.5528665413533835 | Loss: 0.9035863186183729\n",
            "Epoch: 3 | Acc: 0.6764567669172932 | Loss: 0.7448285002457468\n",
            "Epoch: 4 | Acc: 0.7951127819548872 | Loss: 0.5776462068683222\n",
            "Epoch: 5 | Acc: 0.880874060150376 | Loss: 0.45588539462340505\n",
            "Epoch: 6 | Acc: 0.9154135338345865 | Loss: 0.3696721823591935\n",
            "Epoch: 7 | Acc: 0.944078947368421 | Loss: 0.30954146620474365\n",
            "Epoch: 8 | Acc: 0.9649906015037594 | Loss: 0.27081841387246786\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 32  39   1]\n",
            " [ 23 123  15]\n",
            " [  0  32  33]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.58      0.44      0.50        72\n",
            "      medium       0.63      0.76      0.69       161\n",
            "        high       0.67      0.51      0.58        65\n",
            "\n",
            "    accuracy                           0.63       298\n",
            "   macro avg       0.63      0.57      0.59       298\n",
            "weighted avg       0.63      0.63      0.62       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.705\tKappa: 0.35\tAccuracy: 0.631\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5277777777777778 | Loss: 1.039391580555174\n",
            "Epoch: 2 | Acc: 0.5798611111111112 | Loss: 0.9479096697436439\n",
            "Epoch: 3 | Acc: 0.6649305555555556 | Loss: 0.8296046853065491\n",
            "Epoch: 4 | Acc: 0.75 | Loss: 0.7049060066541036\n",
            "Epoch: 5 | Acc: 0.8194444444444444 | Loss: 0.606726747420099\n",
            "Epoch: 6 | Acc: 0.8420138888888888 | Loss: 0.5254058821333779\n",
            "Epoch: 7 | Acc: 0.8940972222222222 | Loss: 0.45551834007104236\n",
            "Epoch: 8 | Acc: 0.890625 | Loss: 0.4465778668721517\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[27 38  4]\n",
            " [30 91 29]\n",
            " [ 9 19 37]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.41      0.39      0.40        69\n",
            "      medium       0.61      0.61      0.61       150\n",
            "        high       0.53      0.57      0.55        65\n",
            "\n",
            "    accuracy                           0.55       284\n",
            "   macro avg       0.52      0.52      0.52       284\n",
            "weighted avg       0.55      0.55      0.55       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.68\tKappa: 0.258\tAccuracy: 0.546\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.704\t Average Kappa: 0.344\t Average Accuracy: 0.637\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DWAPPRUZZG4Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_53_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('MLP1', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP2', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP3', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP4', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP5', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP6', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP7', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP8', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP9', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                                   ('MLP10', MLP(hidden_layer_sizes=(300, 60, )))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k6a2jR7mZZVW",
        "colab_type": "code",
        "outputId": "6ce9122b-5e15-4e39-e61f-42865fc409eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_53_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.553409090909091 | Loss: 0.9618054240942001\n",
            "Epoch: 2 | Acc: 0.6613636363636364 | Loss: 0.8341364622116089\n",
            "Epoch: 3 | Acc: 0.6988636363636364 | Loss: 0.7101561009883881\n",
            "Epoch: 4 | Acc: 0.7596590909090909 | Loss: 0.579218527674675\n",
            "Epoch: 5 | Acc: 0.8 | Loss: 0.47652297765016555\n",
            "Epoch: 6 | Acc: 0.8671875 | Loss: 0.37958798855543135\n",
            "Epoch: 7 | Acc: 0.9063920454545455 | Loss: 0.3280036672949791\n",
            "Epoch: 8 | Acc: 0.9258522727272727 | Loss: 0.28967467322945595\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 18  35   0]\n",
            " [ 15 176  13]\n",
            " [  1  33  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.53      0.34      0.41        53\n",
            "      medium       0.72      0.86      0.79       204\n",
            "        high       0.61      0.37      0.46        54\n",
            "\n",
            "    accuracy                           0.69       311\n",
            "   macro avg       0.62      0.52      0.55       311\n",
            "weighted avg       0.67      0.69      0.67       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.668\tKappa: 0.304\tAccuracy: 0.688\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5344907407407408 | Loss: 1.0029839718783344\n",
            "Epoch: 2 | Acc: 0.5606481481481481 | Loss: 0.9338522178155405\n",
            "Epoch: 3 | Acc: 0.6108796296296296 | Loss: 0.8021419798886334\n",
            "Epoch: 4 | Acc: 0.7331018518518518 | Loss: 0.6088813101803815\n",
            "Epoch: 5 | Acc: 0.8493055555555555 | Loss: 0.4586876156153502\n",
            "Epoch: 6 | Acc: 0.9016203703703703 | Loss: 0.34588843142544784\n",
            "Epoch: 7 | Acc: 0.9210648148148147 | Loss: 0.2790056985837442\n",
            "Epoch: 8 | Acc: 0.95 | Loss: 0.23733846953621618\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 49  43   3]\n",
            " [ 20 191  20]\n",
            " [  0  46  43]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.71      0.52      0.60        95\n",
            "      medium       0.68      0.83      0.75       231\n",
            "        high       0.65      0.48      0.55        89\n",
            "\n",
            "    accuracy                           0.68       415\n",
            "   macro avg       0.68      0.61      0.63       415\n",
            "weighted avg       0.68      0.68      0.67       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.762\tKappa: 0.424\tAccuracy: 0.682\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5169172932330827 | Loss: 1.022900010410108\n",
            "Epoch: 2 | Acc: 0.6292293233082706 | Loss: 0.8783016047979656\n",
            "Epoch: 3 | Acc: 0.6790413533834586 | Loss: 0.7248553821915075\n",
            "Epoch: 4 | Acc: 0.7730263157894737 | Loss: 0.591737006839953\n",
            "Epoch: 5 | Acc: 0.863016917293233 | Loss: 0.47191344123137624\n",
            "Epoch: 6 | Acc: 0.8961466165413534 | Loss: 0.39934985261214406\n",
            "Epoch: 7 | Acc: 0.9222274436090224 | Loss: 0.3352236128167102\n",
            "Epoch: 8 | Acc: 0.9356203007518797 | Loss: 0.31378467459427684\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 29  43   0]\n",
            " [ 29 115  17]\n",
            " [  1  29  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.49      0.40      0.44        72\n",
            "      medium       0.61      0.71      0.66       161\n",
            "        high       0.67      0.54      0.60        65\n",
            "\n",
            "    accuracy                           0.60       298\n",
            "   macro avg       0.59      0.55      0.57       298\n",
            "weighted avg       0.60      0.60      0.59       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.691\tKappa: 0.306\tAccuracy: 0.601\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5190972222222222 | Loss: 1.0271599690119426\n",
            "Epoch: 2 | Acc: 0.5277777777777778 | Loss: 0.979412297407786\n",
            "Epoch: 3 | Acc: 0.5642361111111112 | Loss: 0.8795654012097253\n",
            "Epoch: 4 | Acc: 0.6805555555555556 | Loss: 0.7487395041518741\n",
            "Epoch: 6 | Acc: 0.8368055555555556 | Loss: 0.503231363164054\n",
            "Epoch: 7 | Acc: 0.8836805555555556 | Loss: 0.4218694633907742\n",
            "Epoch: 8 | Acc: 0.9045138888888888 | Loss: 0.38325216869513196\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 24  43   2]\n",
            " [ 16 115  19]\n",
            " [  1  33  31]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.35      0.44        69\n",
            "      medium       0.60      0.77      0.67       150\n",
            "        high       0.60      0.48      0.53        65\n",
            "\n",
            "    accuracy                           0.60       284\n",
            "   macro avg       0.59      0.53      0.55       284\n",
            "weighted avg       0.60      0.60      0.58       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.752\tKappa: 0.293\tAccuracy: 0.599\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.718\t Average Kappa: 0.332\t Average Accuracy: 0.643\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "61YbbuW0BY6g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_54_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('scaler', StandardScaler()),\n",
        "                      ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC4', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC5', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC6', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC7', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC8', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC9', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                ('RFC10', RFC(n_estimators=200, class_weight = 'balanced'))],\n",
        "                                              voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "de-TNNLxCKcp",
        "colab_type": "code",
        "outputId": "0ac1d778-6a6d-4000-ebcf-a7a0f20defc8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_54_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.6504261363636363 | Loss: 0.9127194941043854\n",
            "Epoch: 2 | Acc: 0.6566761363636363 | Loss: 0.8394600093364716\n",
            "Epoch: 3 | Acc: 0.6761363636363636 | Loss: 0.7175031483173371\n",
            "Epoch: 4 | Acc: 0.7448863636363636 | Loss: 0.5816685244441032\n",
            "Epoch: 5 | Acc: 0.8416193181818181 | Loss: 0.46133118122816086\n",
            "Epoch: 6 | Acc: 0.9086647727272726 | Loss: 0.3659888327121735\n",
            "Epoch: 7 | Acc: 0.9242897727272726 | Loss: 0.32052055299282073\n",
            "Epoch: 8 | Acc: 0.9485795454545455 | Loss: 0.28961305767297746\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 21  32   0]\n",
            " [ 12 184   8]\n",
            " [  1  34  19]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.40      0.48        53\n",
            "      medium       0.74      0.90      0.81       204\n",
            "        high       0.70      0.35      0.47        54\n",
            "\n",
            "    accuracy                           0.72       311\n",
            "   macro avg       0.69      0.55      0.59       311\n",
            "weighted avg       0.71      0.72      0.70       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.704\tKappa: 0.363\tAccuracy: 0.72\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5583333333333333 | Loss: 0.9882644436977528\n",
            "Epoch: 2 | Acc: 0.5914351851851852 | Loss: 0.8886253083193744\n",
            "Epoch: 3 | Acc: 0.7152777777777778 | Loss: 0.7308044654351694\n",
            "Epoch: 4 | Acc: 0.8076388888888888 | Loss: 0.5580053649566792\n",
            "Epoch: 5 | Acc: 0.8921296296296296 | Loss: 0.4186569419172075\n",
            "Epoch: 6 | Acc: 0.9141203703703703 | Loss: 0.3417180848342401\n",
            "Epoch: 7 | Acc: 0.9442129629629629 | Loss: 0.2765276128495181\n",
            "Epoch: 8 | Acc: 0.9560185185185185 | Loss: 0.2423191467920939\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 53  38   4]\n",
            " [ 33 186  12]\n",
            " [  2  54  33]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.60      0.56      0.58        95\n",
            "      medium       0.67      0.81      0.73       231\n",
            "        high       0.67      0.37      0.48        89\n",
            "\n",
            "    accuracy                           0.66       415\n",
            "   macro avg       0.65      0.58      0.60       415\n",
            "weighted avg       0.65      0.66      0.64       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.747\tKappa: 0.377\tAccuracy: 0.655\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5103383458646616 | Loss: 1.0170964473172237\n",
            "Epoch: 2 | Acc: 0.5580357142857143 | Loss: 0.8887904066788522\n",
            "Epoch: 3 | Acc: 0.668468045112782 | Loss: 0.7394721131575736\n",
            "Epoch: 4 | Acc: 0.7969924812030075 | Loss: 0.5861597861114302\n",
            "Epoch: 5 | Acc: 0.8738251879699249 | Loss: 0.4737296888702794\n",
            "Epoch: 6 | Acc: 0.9090695488721804 | Loss: 0.3850194231459969\n",
            "Epoch: 7 | Acc: 0.944078947368421 | Loss: 0.3306411802768707\n",
            "Epoch: 8 | Acc: 0.9501879699248119 | Loss: 0.3098003950558211\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 22  49   1]\n",
            " [ 22 125  14]\n",
            " [  0  28  37]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.31      0.38        72\n",
            "      medium       0.62      0.78      0.69       161\n",
            "        high       0.71      0.57      0.63        65\n",
            "\n",
            "    accuracy                           0.62       298\n",
            "   macro avg       0.61      0.55      0.57       298\n",
            "weighted avg       0.61      0.62      0.60       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.731\tKappa: 0.317\tAccuracy: 0.617\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4513888888888889 | Loss: 1.0446637239721086\n",
            "Epoch: 2 | Acc: 0.5520833333333334 | Loss: 0.9238248467445374\n",
            "Epoch: 3 | Acc: 0.625 | Loss: 0.8071493142180972\n",
            "Epoch: 4 | Acc: 0.7291666666666666 | Loss: 0.6833966374397278\n",
            "Epoch: 5 | Acc: 0.8194444444444444 | Loss: 0.5952497091558244\n",
            "Epoch: 6 | Acc: 0.8732638888888888 | Loss: 0.5197618338796828\n",
            "Epoch: 7 | Acc: 0.8923611111111112 | Loss: 0.4691494223144319\n",
            "Epoch: 8 | Acc: 0.90625 | Loss: 0.43297603891955483\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 12  56   1]\n",
            " [ 12 120  18]\n",
            " [  0  34  31]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.17      0.26        69\n",
            "      medium       0.57      0.80      0.67       150\n",
            "        high       0.62      0.48      0.54        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.56      0.48      0.49       284\n",
            "weighted avg       0.57      0.57      0.54       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.709\tKappa: 0.223\tAccuracy: 0.574\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.723\t Average Kappa: 0.32\t Average Accuracy: 0.641\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AUhoq6PuD3ol",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_experiment_55_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('scaler', StandardScaler()),\n",
        "                      ('clf', VotingClassifier([('MLP1', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP2', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP3', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP4', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP5', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP6', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP7', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP8', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP9', MLP(hidden_layer_sizes=(300, 60, ))),\n",
        "                                                ('MLP10', MLP(hidden_layer_sizes=(300, 60, )))],\n",
        "                                              voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xz13IX2IEB9A",
        "colab_type": "code",
        "outputId": "e8cc8b2e-7c23-42c0-ffc9-d4ca4b95c840",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_experiment_55_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.6534090909090909 | Loss: 0.8968979835510253\n",
            "Epoch: 2 | Acc: 0.6583806818181819 | Loss: 0.7686542809009552\n",
            "Epoch: 3 | Acc: 0.7262784090909091 | Loss: 0.6227230727672577\n",
            "Epoch: 4 | Acc: 0.7948863636363637 | Loss: 0.488261017203331\n",
            "Epoch: 5 | Acc: 0.8329545454545455 | Loss: 0.3838814377784729\n",
            "Epoch: 6 | Acc: 0.8992897727272726 | Loss: 0.29928178489208224\n",
            "Epoch: 7 | Acc: 0.9367897727272727 | Loss: 0.2569867707788944\n",
            "Epoch: 8 | Acc: 0.9602272727272727 | Loss: 0.2231905773282051\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 20  32   1]\n",
            " [ 14 178  12]\n",
            " [  0  31  23]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.59      0.38      0.46        53\n",
            "      medium       0.74      0.87      0.80       204\n",
            "        high       0.64      0.43      0.51        54\n",
            "\n",
            "    accuracy                           0.71       311\n",
            "   macro avg       0.66      0.56      0.59       311\n",
            "weighted avg       0.70      0.71      0.69       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.705\tKappa: 0.361\tAccuracy: 0.711\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5543981481481481 | Loss: 0.9874294620973093\n",
            "Epoch: 2 | Acc: 0.5944444444444444 | Loss: 0.8859488743322866\n",
            "Epoch: 3 | Acc: 0.680324074074074 | Loss: 0.7294610429693151\n",
            "Epoch: 4 | Acc: 0.7773148148148148 | Loss: 0.5874160605448263\n",
            "Epoch: 5 | Acc: 0.8710648148148148 | Loss: 0.4309670461548699\n",
            "Epoch: 6 | Acc: 0.8863425925925925 | Loss: 0.35894657726641055\n",
            "Epoch: 7 | Acc: 0.9243055555555556 | Loss: 0.29028211865160203\n",
            "Epoch: 8 | Acc: 0.9386574074074074 | Loss: 0.2573959396945106\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 56  32   7]\n",
            " [ 29 179  23]\n",
            " [  1  47  41]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.59      0.62        95\n",
            "      medium       0.69      0.77      0.73       231\n",
            "        high       0.58      0.46      0.51        89\n",
            "\n",
            "    accuracy                           0.67       415\n",
            "   macro avg       0.64      0.61      0.62       415\n",
            "weighted avg       0.66      0.67      0.66       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.75\tKappa: 0.412\tAccuracy: 0.665\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.49224624060150374 | Loss: 1.022734312634719\n",
            "Epoch: 2 | Acc: 0.5728383458646616 | Loss: 0.9076793382042333\n",
            "Epoch: 3 | Acc: 0.6757518796992481 | Loss: 0.7286377580542314\n",
            "Epoch: 4 | Acc: 0.8113251879699248 | Loss: 0.5594251610730824\n",
            "Epoch: 5 | Acc: 0.8923872180451128 | Loss: 0.4271157666256553\n",
            "Epoch: 6 | Acc: 0.9389097744360902 | Loss: 0.32919066595403773\n",
            "Epoch: 7 | Acc: 0.9504229323308271 | Loss: 0.2624815090706474\n",
            "Epoch: 8 | Acc: 0.9487781954887219 | Loss: 0.24178191313618108\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 26  44   2]\n",
            " [ 25 114  22]\n",
            " [  2  26  37]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.49      0.36      0.42        72\n",
            "      medium       0.62      0.71      0.66       161\n",
            "        high       0.61      0.57      0.59        65\n",
            "\n",
            "    accuracy                           0.59       298\n",
            "   macro avg       0.57      0.55      0.55       298\n",
            "weighted avg       0.59      0.59      0.59       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.708\tKappa: 0.298\tAccuracy: 0.594\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4288194444444444 | Loss: 1.0887904928790197\n",
            "Epoch: 2 | Acc: 0.5486111111111112 | Loss: 0.9404571420616574\n",
            "Epoch: 3 | Acc: 0.6388888888888888 | Loss: 0.8361199332608117\n",
            "Epoch: 4 | Acc: 0.7065972222222222 | Loss: 0.6919092535972595\n",
            "Epoch: 5 | Acc: 0.7465277777777778 | Loss: 0.6084530221091377\n",
            "Epoch: 6 | Acc: 0.7916666666666666 | Loss: 0.5255977577633328\n",
            "Epoch: 7 | Acc: 0.8385416666666666 | Loss: 0.48153814838992226\n",
            "Epoch: 8 | Acc: 0.8333333333333334 | Loss: 0.4643304232094023\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 22  44   3]\n",
            " [ 19 111  20]\n",
            " [  2  34  29]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.51      0.32      0.39        69\n",
            "      medium       0.59      0.74      0.65       150\n",
            "        high       0.56      0.45      0.50        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.55      0.50      0.51       284\n",
            "weighted avg       0.56      0.57      0.55       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.73\tKappa: 0.246\tAccuracy: 0.57\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.723\t Average Kappa: 0.329\t Average Accuracy: 0.635\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVblaGRBN_B2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_final_pipeline():\n",
        "    return Pipeline([('features',\n",
        "                      FeatureUnion([('BERTfeatures', BERTEmbedding()),\n",
        "                                    ('chars_count', CharsCountTransformer()),\n",
        "                                    ('emoji_count', EmojiHandler()),\n",
        "                                    ('liu', LiuFeatureExtractor(tokenizer = TweetTokenizer(preserve_case=False)))\n",
        "                                    ])), ('clf', VotingClassifier([('RFC1', RFC(n_estimators=200, random_state = 1138, class_weight = 'balanced')),\n",
        "                                                                   ('RFC2', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('RFC3', RFC(n_estimators=200, class_weight = 'balanced')),\n",
        "                                                                   ('MLP1', MLP(hidden_layer_sizes=(2000, 500, 100, ), random_state=1138))],\n",
        "                                                                  voting = 'soft'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4xcUmrCeOHZ-",
        "colab_type": "code",
        "outputId": "c8b5fde7-b47f-461f-ac1c-f8284828afdc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = get_final_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.6559659090909091 | Loss: 0.8798013508319855\n",
            "Epoch: 2 | Acc: 0.6637784090909091 | Loss: 0.8162979930639267\n",
            "Epoch: 3 | Acc: 0.7130681818181819 | Loss: 0.7063346058130264\n",
            "Epoch: 4 | Acc: 0.7613636363636364 | Loss: 0.5763058975338936\n",
            "Epoch: 5 | Acc: 0.8055397727272726 | Loss: 0.4657172217965126\n",
            "Epoch: 6 | Acc: 0.8704545454545455 | Loss: 0.3728074200451374\n",
            "Epoch: 7 | Acc: 0.8992897727272726 | Loss: 0.31578714326024054\n",
            "Epoch: 8 | Acc: 0.9328125 | Loss: 0.2740995310246944\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 18  34   1]\n",
            " [ 17 177  10]\n",
            " [  1  33  20]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.50      0.34      0.40        53\n",
            "      medium       0.73      0.87      0.79       204\n",
            "        high       0.65      0.37      0.47        54\n",
            "\n",
            "    accuracy                           0.69       311\n",
            "   macro avg       0.62      0.53      0.56       311\n",
            "weighted avg       0.67      0.69      0.67       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.736\tKappa: 0.311\tAccuracy: 0.691\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5263888888888889 | Loss: 1.0264518702471699\n",
            "Epoch: 2 | Acc: 0.5655092592592593 | Loss: 0.9471615530826427\n",
            "Epoch: 3 | Acc: 0.6171296296296297 | Loss: 0.8478303617901273\n",
            "Epoch: 4 | Acc: 0.6872685185185184 | Loss: 0.7058012949095832\n",
            "Epoch: 5 | Acc: 0.7837962962962963 | Loss: 0.5652810688372012\n",
            "Epoch: 6 | Acc: 0.8472222222222222 | Loss: 0.4374425698209692\n",
            "Epoch: 7 | Acc: 0.8935185185185185 | Loss: 0.3523970224239208\n",
            "Epoch: 8 | Acc: 0.9125 | Loss: 0.31315227294409714\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 56  35   4]\n",
            " [ 21 190  20]\n",
            " [  3  49  37]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.70      0.59      0.64        95\n",
            "      medium       0.69      0.82      0.75       231\n",
            "        high       0.61      0.42      0.49        89\n",
            "\n",
            "    accuracy                           0.68       415\n",
            "   macro avg       0.67      0.61      0.63       415\n",
            "weighted avg       0.68      0.68      0.67       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.787\tKappa: 0.429\tAccuracy: 0.682\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.4953007518796993 | Loss: 0.9965733321089494\n",
            "Epoch: 2 | Acc: 0.5979793233082706 | Loss: 0.8456895131813852\n",
            "Epoch: 3 | Acc: 0.7368421052631579 | Loss: 0.6964677823217291\n",
            "Epoch: 4 | Acc: 0.8352913533834586 | Loss: 0.5529679875624808\n",
            "Epoch: 5 | Acc: 0.8862781954887219 | Loss: 0.4631483256816864\n",
            "Epoch: 6 | Acc: 0.9085996240601504 | Loss: 0.4093131422996521\n",
            "Epoch: 7 | Acc: 0.9302161654135338 | Loss: 0.36641004211024236\n",
            "Epoch: 8 | Acc: 0.9335056390977443 | Loss: 0.33936860059436996\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 32  40   0]\n",
            " [ 20 128  13]\n",
            " [  0  30  35]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.44      0.52        72\n",
            "      medium       0.65      0.80      0.71       161\n",
            "        high       0.73      0.54      0.62        65\n",
            "\n",
            "    accuracy                           0.65       298\n",
            "   macro avg       0.66      0.59      0.62       298\n",
            "weighted avg       0.66      0.65      0.65       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.727\tKappa: 0.387\tAccuracy: 0.654\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.3888888888888889 | Loss: 1.089317974117067\n",
            "Epoch: 2 | Acc: 0.53125 | Loss: 0.9823780722088284\n",
            "Epoch: 3 | Acc: 0.59375 | Loss: 0.8750637802812788\n",
            "Epoch: 4 | Acc: 0.7256944444444444 | Loss: 0.7402433355649313\n",
            "Epoch: 5 | Acc: 0.7934027777777778 | Loss: 0.6255010349882973\n",
            "Epoch: 6 | Acc: 0.8402777777777778 | Loss: 0.5376073320706686\n",
            "Epoch: 7 | Acc: 0.8715277777777778 | Loss: 0.4859101126591365\n",
            "Epoch: 8 | Acc: 0.8836805555555556 | Loss: 0.4525942686531279\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 18  50   1]\n",
            " [ 21 110  19]\n",
            " [  2  30  33]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.44      0.26      0.33        69\n",
            "      medium       0.58      0.73      0.65       150\n",
            "        high       0.62      0.51      0.56        65\n",
            "\n",
            "    accuracy                           0.57       284\n",
            "   macro avg       0.55      0.50      0.51       284\n",
            "weighted avg       0.55      0.57      0.55       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.702\tKappa: 0.239\tAccuracy: 0.567\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.738\t Average Kappa: 0.342\t Average Accuracy: 0.649\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-07T15:44:21.384119Z",
          "start_time": "2020-04-07T15:44:21.170488Z"
        },
        "scrolled": false,
        "id": "U1WXkA-9Q1r2",
        "colab_type": "code",
        "outputId": "f8c1485e-26b6-4290-8d3d-6132771c8cb7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    #pipeline = get_experiment_0_pipeline()\n",
        "    #pipeline = ELMo_pipeline()\n",
        "    #pipeline = BERT_pipeline()\n",
        "    pipeline = RFs_pipeline()\n",
        "    #pipeline = RFs_bal_pipeline()\n",
        "    #pipeline = MLPs_pipeline()\n",
        "    #pipeline = RFs_MLPs_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "    #classifier, learned_labels, scores = run_test(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "print(\n",
        "    \"Average scores:\\n\\n\",\n",
        "    \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.653125 | Loss: 0.8947009712457656\n",
            "Epoch: 2 | Acc: 0.6559659090909091 | Loss: 0.8092648774385452\n",
            "Epoch: 3 | Acc: 0.6754261363636364 | Loss: 0.7005722105503083\n",
            "Epoch: 4 | Acc: 0.720028409090909 | Loss: 0.585888808965683\n",
            "Epoch: 5 | Acc: 0.7910511363636363 | Loss: 0.49063661247491835\n",
            "Epoch: 6 | Acc: 0.8798295454545455 | Loss: 0.4126951828598976\n",
            "Epoch: 7 | Acc: 0.9025568181818182 | Loss: 0.3861504182219505\n",
            "Delete\n",
            "Confusion Matrix for anger:\n",
            "\n",
            "[[ 17  36   0]\n",
            " [ 11 185   8]\n",
            " [  2  30  22]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.57      0.32      0.41        53\n",
            "      medium       0.74      0.91      0.81       204\n",
            "        high       0.73      0.41      0.52        54\n",
            "\n",
            "    accuracy                           0.72       311\n",
            "   macro avg       0.68      0.55      0.58       311\n",
            "weighted avg       0.71      0.72      0.69       311\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.745\tKappa: 0.36\tAccuracy: 0.72\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5453703703703704 | Loss: 0.9779080329117952\n",
            "Epoch: 2 | Acc: 0.6212962962962962 | Loss: 0.8082942719812747\n",
            "Epoch: 3 | Acc: 0.7587962962962963 | Loss: 0.6283684185257664\n",
            "Epoch: 4 | Acc: 0.8444444444444444 | Loss: 0.4777123111265677\n",
            "Epoch: 5 | Acc: 0.9178240740740741 | Loss: 0.3599268567782861\n",
            "Epoch: 6 | Acc: 0.9370370370370371 | Loss: 0.29390310247739154\n",
            "Epoch: 7 | Acc: 0.9465277777777777 | Loss: 0.25899169952781115\n",
            "Delete\n",
            "Confusion Matrix for fear:\n",
            "\n",
            "[[ 46  49   0]\n",
            " [ 25 187  19]\n",
            " [  0  36  53]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.65      0.48      0.55        95\n",
            "      medium       0.69      0.81      0.74       231\n",
            "        high       0.74      0.60      0.66        89\n",
            "\n",
            "    accuracy                           0.69       415\n",
            "   macro avg       0.69      0.63      0.65       415\n",
            "weighted avg       0.69      0.69      0.68       415\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.792\tKappa: 0.444\tAccuracy: 0.689\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5284304511278195 | Loss: 1.0055534776888395\n",
            "Epoch: 2 | Acc: 0.6033834586466165 | Loss: 0.8500318935042933\n",
            "Epoch: 3 | Acc: 0.7415413533834586 | Loss: 0.6878674469496074\n",
            "Epoch: 4 | Acc: 0.8498590225563909 | Loss: 0.5396466380671451\n",
            "Epoch: 5 | Acc: 0.8860432330827067 | Loss: 0.4519508073204442\n",
            "Epoch: 6 | Acc: 0.9172932330827067 | Loss: 0.3687887505481118\n",
            "Epoch: 7 | Acc: 0.9419642857142856 | Loss: 0.33166033186410604\n",
            "Delete\n",
            "Confusion Matrix for joy:\n",
            "\n",
            "[[ 31  40   1]\n",
            " [ 19 130  12]\n",
            " [  0  24  41]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.62      0.43      0.51        72\n",
            "      medium       0.67      0.81      0.73       161\n",
            "        high       0.76      0.63      0.69        65\n",
            "\n",
            "    accuracy                           0.68       298\n",
            "   macro avg       0.68      0.62      0.64       298\n",
            "weighted avg       0.68      0.68      0.67       298\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.81\tKappa: 0.433\tAccuracy: 0.678\n",
            "------------------------------------------------------\n",
            "\n",
            "Training BERT embeddings...\n",
            "Epoch: 1 | Acc: 0.5190972222222222 | Loss: 1.0235905945301056\n",
            "Epoch: 2 | Acc: 0.5347222222222222 | Loss: 0.9388632443216112\n",
            "Epoch: 3 | Acc: 0.6215277777777778 | Loss: 0.8379210531711578\n",
            "Epoch: 4 | Acc: 0.6857638888888888 | Loss: 0.712523361047109\n",
            "Epoch: 5 | Acc: 0.7447916666666666 | Loss: 0.6247052119837867\n",
            "Epoch: 6 | Acc: 0.7777777777777778 | Loss: 0.5601390252510706\n",
            "Epoch: 7 | Acc: 0.8003472222222222 | Loss: 0.5214650796519386\n",
            "Delete\n",
            "Confusion Matrix for sadness:\n",
            "\n",
            "[[ 25  43   1]\n",
            " [ 16 118  16]\n",
            " [  0  36  29]]\n",
            "\n",
            "Classification Report:\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "         low       0.61      0.36      0.45        69\n",
            "      medium       0.60      0.79      0.68       150\n",
            "        high       0.63      0.45      0.52        65\n",
            "\n",
            "    accuracy                           0.61       284\n",
            "   macro avg       0.61      0.53      0.55       284\n",
            "weighted avg       0.61      0.61      0.59       284\n",
            "\n",
            "Scores:\n",
            "\n",
            "AUC:  0.743\tKappa: 0.298\tAccuracy: 0.606\n",
            "------------------------------------------------------\n",
            "\n",
            "Average scores:\n",
            "\n",
            " Average AUC: 0.772\t Average Kappa: 0.384\t Average Accuracy: 0.673\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-07T15:44:21.392097Z",
          "start_time": "2020-04-07T15:44:21.386114Z"
        },
        "id": "4LiZ9bpRQ1r6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def predict_target(dataset, classifier, labels):\n",
        "    # Predecir las probabilidades de intensidad de cada elemento del target set.\n",
        "    predicted = pd.DataFrame(classifier.predict_proba(dataset.tweet), columns=labels)\n",
        "    # Agregar ids\n",
        "    predicted['id'] = dataset.id.values\n",
        "    # Reordenar las columnas\n",
        "    predicted = predicted[['id', 'low', 'medium', 'high']]\n",
        "    return predicted"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-04-07T15:44:21.588573Z",
          "start_time": "2020-04-07T15:44:21.394094Z"
        },
        "scrolled": true,
        "id": "_J9e3FX6Q1r8",
        "colab_type": "code",
        "outputId": "5c0f774c-6aee-4845-eff6-cbbbb161bd1c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "source": [
        "predicted_target = {}\n",
        "\n",
        "# Crear carpeta ./predictions\n",
        "if (not os.path.exists('./predictions')):\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "else:\n",
        "    # Eliminar predicciones anteriores:\n",
        "    shutil.rmtree('./predictions')\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "# por cada target set:\n",
        "for idx, key in enumerate(target):\n",
        "    # Predecirlo\n",
        "    predicted_target[key] = predict_target(target[key], classifiers[idx],\n",
        "                                           learned_labels_array[idx])\n",
        "    # Guardar predicciones en archivos separados. \n",
        "    predicted_target[key].to_csv('./predictions/{}-pred.txt'.format(key),\n",
        "                                 sep='\\t',\n",
        "                                 header=False,\n",
        "                                 index=False)\n",
        "\n",
        "# Crear archivo zip\n",
        "a = shutil.make_archive('predictions', 'zip', './predictions')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Delete\n",
            "Delete\n",
            "Delete\n",
            "Delete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOtwX-qdbG_I",
        "colab_type": "text"
      },
      "source": [
        "Habiendo elegido el modelo este se entrena con todo el conjunto de entrenamiento, para aprovechar los datos no utilizados previamente."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JMYvJOVkOXM1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def whole_run(dataset, dataset_name, pipeline):\n",
        "    \"\"\"Creamos el pipeline y luego lo ejecutamos el pipeline sobre un dataset. \n",
        "    Retorna el modelo ya entrenado mas sus labels asociadas y los scores obtenidos al evaluarlo.\"\"\"\n",
        "\n",
        "    # Dividimos el dataset en train y test.\n",
        "    # X_train, X_test, y_train, y_test = train_test_split(\n",
        "    #     dataset.tweet,\n",
        "    #     dataset.sentiment_intensity,\n",
        "    #     random_state = 1138,\n",
        "    #     shuffle=True,\n",
        "    #     test_size=0.33)\n",
        "\n",
        "    # Entrenamos el clasificador (Ejecuta el entrenamiento sobre todo el pipeline)\n",
        "    #pipeline.fit(X_train, y_train)\n",
        "    pipeline.fit(dataset.tweet, dataset.sentiment_intensity)\n",
        "\n",
        "    # Predecimos las probabilidades de intensidad de cada elemento del set de prueba.\n",
        "    #predicted_probabilities = pipeline.predict_proba(X_test)\n",
        "\n",
        "    # Obtenemos el orden de las clases aprendidas.\n",
        "    learned_labels = pipeline.classes_\n",
        "\n",
        "    # Evaluamos:\n",
        "    #scores = evaulate(predicted_probabilities, y_test, learned_labels, dataset_name)\n",
        "    return pipeline, learned_labels, #scores"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x5Ks9gHCQBHG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "classifiers = []\n",
        "learned_labels_array = []\n",
        "scores_array = []\n",
        "\n",
        "# Por cada nombre_dataset, dataset en train ('anger', 'fear', 'joy', 'sadness')\n",
        "for dataset_name, dataset in train.items():\n",
        "    \n",
        "    # creamos el pipeline\n",
        "    pipeline = RFs_pipeline()\n",
        "    \n",
        "    # ejecutamos el pipeline sobre el dataset\n",
        "    #classifier, learned_labels, scores = run(dataset, dataset_name, pipeline)\n",
        "    classifier, learned_labels = whole_run(dataset, dataset_name, pipeline)\n",
        "\n",
        "    # guardamos el clasificador entrenado (en realidad es el pipeline ya entrenado...)\n",
        "    classifiers.append(classifier)\n",
        "\n",
        "    # guardamos las labels aprendidas por el clasificador\n",
        "    learned_labels_array.append(learned_labels)\n",
        "\n",
        "    # guardamos los scores obtenidos\n",
        "    #scores_array.append(scores)\n",
        "\n",
        "# print avg scores\n",
        "# print(\n",
        "#     \"Average scores:\\n\\n\",\n",
        "#     \"Average AUC: {0:.3g}\\t Average Kappa: {1:.3g}\\t Average Accuracy: {2:.3g}\"\n",
        "#    .format(*np.array(scores_array).mean(axis=0)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "agWpuQpqPbzJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "predicted_target = {}\n",
        "\n",
        "# Crear carpeta ./predictions\n",
        "if (not os.path.exists('./predictions')):\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "else:\n",
        "    # Eliminar predicciones anteriores:\n",
        "    shutil.rmtree('./predictions')\n",
        "    os.mkdir('./predictions')\n",
        "\n",
        "# por cada target set:\n",
        "for idx, key in enumerate(target):\n",
        "    # Predecirlo\n",
        "    predicted_target[key] = predict_target(target[key], classifiers[idx],\n",
        "                                           learned_labels_array[idx])\n",
        "    # Guardar predicciones en archivos separados. \n",
        "    predicted_target[key].to_csv('./predictions/{}-pred.txt'.format(key),\n",
        "                                 sep='\\t',\n",
        "                                 header=False,\n",
        "                                 index=False)\n",
        "\n",
        "# Crear archivo zip\n",
        "a = shutil.make_archive('predictionsx', 'zip', './predictions')"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}