06: Zero_Shot_Classification_with_HuggingFace_Pipeline.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Zero_Shot_Classification_with_HuggingFace_Pipeline_4.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gulabpatel/NLP_transformers/blob/main/Zero_Shot_Classification_with_HuggingFace_Pipeline_4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yTBP_QYuu6tc"
      },
      "source": [
        "!pip install transformers==3.1.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TiU_ES5tzpMH"
      },
      "source": [
        "from transformers import pipeline"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "spkccRiv0CB3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0344ae1d-6588-4ccf-e0eb-f3d44955b3bf"
      },
      "source": [
        "classifier = pipeline(\"zero-shot-classification\")\n",
        "# classifier = pipeline(\"zero-shot-classification\", device=0) # to utilize GPU"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartForSequenceClassification: ['model.encoder.version', 'model.decoder.version']\n",
            "- This IS expected if you are initializing BartForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
            "- This IS NOT expected if you are initializing BartForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWiovVJG9ei_"
      },
      "source": [
        "We can use this pipeline by passing in a sequence and a list of candidate labels. The pipeline assumes by default that only one of the candidate labels is true, returning a list of scores for each label which add up to 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hkfE6NRA0Dzy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "977ad530-295c-4f55-eea3-65f234e35350"
      },
      "source": [
        "sequence = \"Who are you voting for in 2020?\"\n",
        "candidate_labels = [\"politics\", \"public health\", \"economics\"]\n",
        "\n",
        "classifier(sequence, candidate_labels)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['politics', 'economics', 'public health'],\n",
              " 'scores': [0.972518801689148, 0.014584142714738846, 0.012897025793790817],\n",
              " 'sequence': 'Who are you voting for in 2020?'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PGXwxxyn9nOC"
      },
      "source": [
        "To do multi-class classification, simply pass `multi_class=True`. In this case, the scores will be independent, but each will fall between 0 and 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZvZeVb2h5RX0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61bb25e1-ded0-4917-fde4-a69402e496de"
      },
      "source": [
        "sequence = \"Who are you voting for in 2020?\"\n",
        "candidate_labels = [\"politics\", \"public health\", \"economics\", \"elections\"]\n",
        "\n",
        "classifier(sequence, candidate_labels, multi_class=True)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['politics', 'elections', 'public health', 'economics'],\n",
              " 'scores': [0.972069501876831,\n",
              "  0.967610776424408,\n",
              "  0.03248710557818413,\n",
              "  0.0061644683592021465],\n",
              " 'sequence': 'Who are you voting for in 2020?'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lLLeDT1r9-yQ"
      },
      "source": [
        "Here's an example of sentiment classification: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f7AF53Wl5f8W",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "42856a6c-d75a-4d03-9fd3-9cd269d5509b"
      },
      "source": [
        "sequence = \"I hated this movie. The acting sucked.\"\n",
        "candidate_labels = [\"positive\", \"negative\"]\n",
        "\n",
        "classifier(sequence, candidate_labels)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['negative', 'positive'],\n",
              " 'scores': [0.9916268587112427, 0.00837317667901516],\n",
              " 'sequence': 'I hated this movie. The acting sucked.'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uSoBpCpV6k4s"
      },
      "source": [
        "So how does this method work?\n",
        "\n",
        "The underlying model is trained on the task of Natural Language Inference (NLI), which takes in two sequences and determines whether they contradict each other, entail each other, or neither.\n",
        "\n",
        "This can be adapted to the task of zero-shot classification by treating the sequence which we want to classify as one NLI sequence (called the premise) and turning a candidate label into the other (the hypothesis). If the model predicts that the constructed premise _entails_ the hypothesis, then we can take that as a prediction that the label applies to the text. Check out [this blog post](https://joeddav.github.io/blog/2020/05/29/ZSL.html) for a more detailed explanation.\n",
        "\n",
        "By default, the pipeline turns labels into hypotheses with the template `This example is {class_name}.`. This works well in many settings, but you can also customize this for your specific setting. Let's add another review to our above sentiment classification example that's a bit more challenging:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5yLx3pRr5xQA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5fea0d95-d6c2-4a91-c1fb-2802d3f034af"
      },
      "source": [
        "sequences = [\n",
        "    \"I hated this movie. The acting sucked.\",\n",
        "    \"This movie didn't quite live up to my high expectations, but overall I still really enjoyed it.\"\n",
        "]\n",
        "candidate_labels = [\"positive\", \"negative\"]\n",
        "\n",
        "classifier(sequences, candidate_labels)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'labels': ['negative', 'positive'],\n",
              "  'scores': [0.9916267991065979, 0.008373182266950607],\n",
              "  'sequence': 'I hated this movie. The acting sucked.'},\n",
              " {'labels': ['negative', 'positive'],\n",
              "  'scores': [0.8148515820503235, 0.1851484179496765],\n",
              "  'sequence': \"This movie didn't quite live up to my high expectations, but overall I still really enjoyed it.\"}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CfrpyGWM782R"
      },
      "source": [
        "The second example is a bit harder. Let's see if we can improve the results by using a hypothesis template which is more specific to the setting of review sentiment analysis. Instead of the default, `This example is {}.`, we'll use, `The sentiment of this review is {}.` (where `{}` is replaced with the candidate class name)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kqx5hp7X8XNA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1b8bcfa4-fa8a-4f3c-eb36-d30f8d5a7597"
      },
      "source": [
        "sequences = [\n",
        "    \"I hated this movie. The acting sucked.\",\n",
        "    \"This movie didn't quite live up to my high expectations, but overall I still really enjoyed it.\"\n",
        "]\n",
        "candidate_labels = [\"positive\", \"negative\"]\n",
        "hypothesis_template = \"The sentiment of this review is {}.\"\n",
        "\n",
        "classifier(sequences, candidate_labels, hypothesis_template=hypothesis_template)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'labels': ['negative', 'positive'],\n",
              "  'scores': [0.9890093207359314, 0.01099067460745573],\n",
              "  'sequence': 'I hated this movie. The acting sucked.'},\n",
              " {'labels': ['positive', 'negative'],\n",
              "  'scores': [0.9581228494644165, 0.0418771356344223],\n",
              "  'sequence': \"This movie didn't quite live up to my high expectations, but overall I still really enjoyed it.\"}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iArbRAe781-_"
      },
      "source": [
        "By providing a more precise hypothesis template, we are able to see a more accurate classification of the second review.\n",
        "\n",
        "> Note that sentiment classification is used here just as an illustrative example. The [Hugging Face Model Hub](https://huggingface.co/models?filter=text-classification) has a number of models trained specifically on sentiment tasks which can be used instead."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XxUTOnllSH4w"
      },
      "source": [
        "#### Update: Zero-shot classification in 100 languages\n",
        "\n",
        "Interested in using the pipeline for languages other than English? We've trained a cross-lingual model on top of XLM RoBERTa which you can use by passing `model='joeddav/xlm-roberta-large-xnli'` when creating the pipeline:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "siZhFPekSN7t"
      },
      "source": [
        "classifier = pipeline(\"zero-shot-classification\", model='joeddav/xlm-roberta-large-xnli')"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zrcljZ75UxKN"
      },
      "source": [
        "You can use it with any combination of languages. For example, let's classify a Russian sentence with English candidate labels:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gBJyFwC2TwGv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4b48d41e-9f3f-40cd-eb25-08a212aa9126"
      },
      "source": [
        "sequence = \"За кого вы голосуете в 2020 году?\" # translation: \"Who are you voting for in 2020?\"\n",
        "candidate_labels = [\"Europe\", \"public health\", \"politics\"]\n",
        "\n",
        "classifier(sequence, candidate_labels)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['politics', 'Europe', 'public health'],\n",
              " 'scores': [0.9048484563827515, 0.05722189322113991, 0.03792969509959221],\n",
              " 'sequence': 'За кого вы голосуете в 2020 году?'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9hGVMsrVI8S"
      },
      "source": [
        "Now let's do the same but with the labels in French:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0xKYBOLYVeNJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "46f2e0fe-4d23-4715-b5a4-0ca5191e3f87"
      },
      "source": [
        "sequence = \"За кого вы голосуете в 2020 году?\" # translation: \"Who are you voting for in 2020?\"\n",
        "candidate_labels = [\"Europe\", \"santé publique\", \"politique\"]\n",
        "\n",
        "classifier(sequence, candidate_labels)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['politique', 'Europe', 'santé publique'],\n",
              " 'scores': [0.9726154804229736, 0.017128491774201393, 0.010256024077534676],\n",
              " 'sequence': 'За кого вы голосуете в 2020 году?'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EHURJUPCVgGP"
      },
      "source": [
        "As we discussed in the last section, the default hypothesis template is the English, `This text is {}.`. If you are working strictly within one language, it may be worthwhile to translate this to the language you are working with:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZCtTclt7VpMv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e396e7a-6b14-4c12-fd70-0305e1ae2f45"
      },
      "source": [
        "sequence = \"¿A quién vas a votar en 2020?\"\n",
        "candidate_labels = [\"Europa\", \"salud pública\", \"política\"]\n",
        "hypothesis_template = \"Este ejemplo es {}.\"\n",
        "\n",
        "classifier(sequence, candidate_labels, hypothesis_template=hypothesis_template)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'labels': ['política', 'Europa', 'salud pública'],\n",
              " 'scores': [0.9109586477279663, 0.05954807624220848, 0.029493309557437897],\n",
              " 'sequence': '¿A quién vas a votar en 2020?'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OQyVNE2fTDVs"
      },
      "source": [
        "The model is fine-tuned on XNLI which includes 15 languages: Arabic, Bulgarian, Chinese, English, French, German, Greek, Hindi, Russian, Spanish, Swahili, Thai, Turkish, Urdu, and Vietnamese. The base model is trained on 85 more, so the model will work to some degree for any of those in the XLM RoBERTa training corpus (see the full list in appendix A of the [XLM Roberata paper](https://arxiv.org/abs/1911.02116)).\n",
        "\n",
        "See the [model page](https://huggingface.co/joeddav/xlm-roberta-large-xnli) for more."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSPWtMo0094O"
      },
      "source": [
        "------------------------------------------------"
      ]
    }
  ]
}