Named_Entity_Recognition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Named-Entity-Recognition.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.7"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Shantanu9326/Data-Science-Portfolio/blob/master/Named_Entity_Recognition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NT0fNtYLFyP8",
        "colab_type": "text"
      },
      "source": [
        "# Named Entity Recognition (NER)\n",
        "spaCy has an **'ner'** pipeline component that identifies token spans fitting a predetermined set of named entities. These are available as the `ents` property of a `Doc` object."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ec_QhKxTGCFR",
        "colab_type": "code",
        "outputId": "3b998edf-78d5-4f41-f2e0-597b712971e4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 390
        }
      },
      "source": [
        "pip install -U spacy[cuda92]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already up-to-date: spacy[cuda92] in /usr/local/lib/python3.6/dist-packages (2.1.4)\n",
            "Requirement already satisfied, skipping upgrade: srsly<1.1.0,>=0.0.5 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (0.0.7)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (1.16.4)\n",
            "Requirement already satisfied, skipping upgrade: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (1.0.2)\n",
            "Requirement already satisfied, skipping upgrade: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (0.9.6)\n",
            "Requirement already satisfied, skipping upgrade: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (2.21.0)\n",
            "Requirement already satisfied, skipping upgrade: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (0.2.4)\n",
            "Requirement already satisfied, skipping upgrade: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (2.0.2)\n",
            "Requirement already satisfied, skipping upgrade: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (0.2.2)\n",
            "Requirement already satisfied, skipping upgrade: thinc<7.1.0,>=7.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (7.0.4)\n",
            "Requirement already satisfied, skipping upgrade: jsonschema<3.1.0,>=2.6.0 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (2.6.0)\n",
            "Requirement already satisfied, skipping upgrade: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (2.0.1)\n",
            "Requirement already satisfied, skipping upgrade: thinc-gpu-ops<0.1.0,>=0.0.1; extra == \"cuda92\" in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (0.0.4)\n",
            "Requirement already satisfied, skipping upgrade: cupy-cuda92>=5.0.0b4; extra == \"cuda92\" in /usr/local/lib/python3.6/dist-packages (from spacy[cuda92]) (7.0.0b1)\n",
            "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda92]) (2.8)\n",
            "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda92]) (3.0.4)\n",
            "Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda92]) (1.24.3)\n",
            "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy[cuda92]) (2019.6.16)\n",
            "Requirement already satisfied, skipping upgrade: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.6/dist-packages (from thinc<7.1.0,>=7.0.2->spacy[cuda92]) (4.28.1)\n",
            "Requirement already satisfied, skipping upgrade: fastrlock>=0.3 in /usr/local/lib/python3.6/dist-packages (from cupy-cuda92>=5.0.0b4; extra == \"cuda92\"->spacy[cuda92]) (0.4)\n",
            "Requirement already satisfied, skipping upgrade: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from cupy-cuda92>=5.0.0b4; extra == \"cuda92\"->spacy[cuda92]) (1.12.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M655KUdkFyP9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Perform standard imports\n",
        "import spacy\n",
        "spacy.prefer_gpu()\n",
        "nlp = spacy.load('en_core_web_sm')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I_FDKDZzFyQA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Write a function to display basic entity info:\n",
        "def show_ents(doc):\n",
        "    if doc.ents:\n",
        "        for ent in doc.ents:\n",
        "            print(ent.text+' - '+ent.label_+' - '+str(spacy.explain(ent.label_)))\n",
        "    else:\n",
        "        print('No named entities found.')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "GOCYs6wKFyQC",
        "colab_type": "code",
        "outputId": "1e2452e5-5289-47ef-8c9e-573a30c04041",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "doc = nlp(u'May I go to Washington, DC next May to see the Washington Monument?')\n",
        "\n",
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Washington, DC - GPE - Countries, cities, states\n",
            "next May - DATE - Absolute or relative dates or periods\n",
            "the Washington Monument - ORG - Companies, agencies, institutions, etc.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_yYw2u_MFyQG",
        "colab_type": "text"
      },
      "source": [
        "Here we see tokens combine to form the entities `Washington, DC`, `next May` and `the Washington Monument`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a9NQ3CWWFyQH",
        "colab_type": "text"
      },
      "source": [
        "## Entity annotations\n",
        "`Doc.ents` are token spans with their own set of annotations.\n",
        "<table>\n",
        "<tr><td>`ent.text`</td><td>The original entity text</td></tr>\n",
        "<tr><td>`ent.label`</td><td>The entity type's hash value</td></tr>\n",
        "<tr><td>`ent.label_`</td><td>The entity type's string description</td></tr>\n",
        "<tr><td>`ent.start`</td><td>The token span's *start* index position in the Doc</td></tr>\n",
        "<tr><td>`ent.end`</td><td>The token span's *stop* index position in the Doc</td></tr>\n",
        "<tr><td>`ent.start_char`</td><td>The entity text's *start* index position in the Doc</td></tr>\n",
        "<tr><td>`ent.end_char`</td><td>The entity text's *stop* index position in the Doc</td></tr>\n",
        "</table>\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_cP9qnCEFyQI",
        "colab_type": "code",
        "outputId": "5eeaf018-d62d-4d17-fc30-7425e55ad6cc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "doc = nlp(u'Can I please borrow 500 dollars from you to buy some Microsoft stock?')\n",
        "\n",
        "for ent in doc.ents:\n",
        "    print(ent.text, ent.start, ent.end, ent.start_char, ent.end_char, ent.label_)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "500 dollars 4 6 20 31 MONEY\n",
            "Microsoft 11 12 53 62 ORG\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bqvuj6byFyQK",
        "colab_type": "text"
      },
      "source": [
        "## NER Tags\n",
        "Tags are accessible through the `.label_` property of an entity.\n",
        "<table>\n",
        "<tr><th>TYPE</th><th>DESCRIPTION</th><th>EXAMPLE</th></tr>\n",
        "<tr><td>`PERSON`</td><td>People, including fictional.</td><td>*Fred Flintstone*</td></tr>\n",
        "<tr><td>`NORP`</td><td>Nationalities or religious or political groups.</td><td>*The Republican Party*</td></tr>\n",
        "<tr><td>`FAC`</td><td>Buildings, airports, highways, bridges, etc.</td><td>*Logan International Airport, The Golden Gate*</td></tr>\n",
        "<tr><td>`ORG`</td><td>Companies, agencies, institutions, etc.</td><td>*Microsoft, FBI, MIT*</td></tr>\n",
        "<tr><td>`GPE`</td><td>Countries, cities, states.</td><td>*France, UAR, Chicago, Idaho*</td></tr>\n",
        "<tr><td>`LOC`</td><td>Non-GPE locations, mountain ranges, bodies of water.</td><td>*Europe, Nile River, Midwest*</td></tr>\n",
        "<tr><td>`PRODUCT`</td><td>Objects, vehicles, foods, etc. (Not services.)</td><td>*Formula 1*</td></tr>\n",
        "<tr><td>`EVENT`</td><td>Named hurricanes, battles, wars, sports events, etc.</td><td>*Olympic Games*</td></tr>\n",
        "<tr><td>`WORK_OF_ART`</td><td>Titles of books, songs, etc.</td><td>*The Mona Lisa*</td></tr>\n",
        "<tr><td>`LAW`</td><td>Named documents made into laws.</td><td>*Roe v. Wade*</td></tr>\n",
        "<tr><td>`LANGUAGE`</td><td>Any named language.</td><td>*English*</td></tr>\n",
        "<tr><td>`DATE`</td><td>Absolute or relative dates or periods.</td><td>*20 July 1969*</td></tr>\n",
        "<tr><td>`TIME`</td><td>Times smaller than a day.</td><td>*Four hours*</td></tr>\n",
        "<tr><td>`PERCENT`</td><td>Percentage, including \"%\".</td><td>*Eighty percent*</td></tr>\n",
        "<tr><td>`MONEY`</td><td>Monetary values, including unit.</td><td>*Twenty Cents*</td></tr>\n",
        "<tr><td>`QUANTITY`</td><td>Measurements, as of weight or distance.</td><td>*Several kilometers, 55kg*</td></tr>\n",
        "<tr><td>`ORDINAL`</td><td>\"first\", \"second\", etc.</td><td>*9th, Ninth*</td></tr>\n",
        "<tr><td>`CARDINAL`</td><td>Numerals that do not fall under another type.</td><td>*2, Two, Fifty-two*</td></tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xb0zcaWpFyQL",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Adding a Named Entity to a Span\n",
        "Normally we would have spaCy build a library of named entities by training it on several samples of text.<br>In this case, we only want to add one value:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9iNiaZL6FyQL",
        "colab_type": "code",
        "outputId": "19d9802e-76d2-49f7-fb01-bfbee6f43336",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "doc = nlp(u'Tesla to build a U.K. factory for $6 million')\n",
        "\n",
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "U.K. - GPE - Countries, cities, states\n",
            "$6 million - MONEY - Monetary values, including unit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AgW5MSwFFyQN",
        "colab_type": "text"
      },
      "source": [
        "<font color=green>Right now, spaCy does not recognize \"Tesla\" as a company.</font>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ce6dZ1kJFyQO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from spacy.tokens import Span\n",
        "\n",
        "# Get the hash value of the ORG entity label\n",
        "ORG = doc.vocab.strings[u'ORG']  \n",
        "\n",
        "# Create a Span for the new entity\n",
        "new_ent = Span(doc, 0, 1, label=ORG)\n",
        "\n",
        "# Add the entity to the existing Doc object\n",
        "doc.ents = list(doc.ents) + [new_ent]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1VysmzTFyQP",
        "colab_type": "text"
      },
      "source": [
        "<font color=green>In the code above, the arguments passed to `Span()` are:</font>\n",
        "-  `doc` - the name of the Doc object\n",
        "-  `0` - the *start* index position of the span\n",
        "-  `1` - the *stop* index position (exclusive)\n",
        "-  `label=ORG` - the label assigned to our entity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "Qbkdml1RFyQQ",
        "colab_type": "code",
        "outputId": "9f69cf94-7ec5-46ea-8b57-7fb7c512e239",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tesla - ORG - Companies, agencies, institutions, etc.\n",
            "U.K. - GPE - Countries, cities, states\n",
            "$6 million - MONEY - Monetary values, including unit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tL1UcSS3FyQS",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Adding Named Entities to All Matching Spans\n",
        "What if we want to tag *all* occurrences of \"Tesla\"? In this section we show how to use the PhraseMatcher to identify a series of spans in the Doc:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jMQSN9SNFyQS",
        "colab_type": "code",
        "outputId": "693d31dc-5697-4407-d64d-0f015dcbd656",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "doc = nlp(u'Our company plans to introduce a new vacuum cleaner. '\n",
        "          u'If successful, the vacuum cleaner will be our first product.')\n",
        "\n",
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "first - ORDINAL - \"first\", \"second\", etc.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UXbWFRgSFyQV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import PhraseMatcher and create a matcher object:\n",
        "from spacy.matcher import PhraseMatcher\n",
        "matcher = PhraseMatcher(nlp.vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x6gDIZNCFyQW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create the desired phrase patterns:\n",
        "phrase_list = ['vacuum cleaner', 'vacuum-cleaner']\n",
        "phrase_patterns = [nlp(text) for text in phrase_list]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wZMuaZ7mFyQZ",
        "colab_type": "code",
        "outputId": "d6f5c7be-a7a7-4217-8d50-119b35c2c6db",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Apply the patterns to our matcher object:\n",
        "matcher.add('newproduct', None, *phrase_patterns)\n",
        "\n",
        "# Apply the matcher to our Doc object:\n",
        "matches = matcher(doc)\n",
        "\n",
        "# See what matches occur:\n",
        "matches"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(2689272359382549672, 7, 9), (2689272359382549672, 14, 16)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F_1UGfzMFyQb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Here we create Spans from each match, and create named entities from them:\n",
        "from spacy.tokens import Span\n",
        "\n",
        "PROD = doc.vocab.strings[u'PRODUCT']\n",
        "\n",
        "new_ents = [Span(doc, match[1],match[2],label=PROD) for match in matches]\n",
        "\n",
        "doc.ents = list(doc.ents) + new_ents"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-18PVOtqFyQe",
        "colab_type": "code",
        "outputId": "f1ccd39f-38ed-4439-89f0-70a79523591f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "vacuum cleaner - PRODUCT - Objects, vehicles, foods, etc. (not services)\n",
            "vacuum cleaner - PRODUCT - Objects, vehicles, foods, etc. (not services)\n",
            "first - ORDINAL - \"first\", \"second\", etc.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RP3XqcZFyQg",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Counting Entities\n",
        "While spaCy may not have a built-in tool for counting entities, we can pass a conditional statement into a list comprehension:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1RDvY1YuFyQh",
        "colab_type": "code",
        "outputId": "2def8844-f93c-40a6-ffb4-9d856d703ca4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "doc = nlp(u'Originally priced at $29.50, the sweater was marked down to five dollars.')\n",
        "\n",
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "29.50 - MONEY - Monetary values, including unit\n",
            "five dollars - MONEY - Monetary values, including unit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s6wMXGbFFyQk",
        "colab_type": "code",
        "outputId": "5df7579c-8c82-49b3-b444-8ee690b47f06",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "len([ent for ent in doc.ents if ent.label_=='MONEY'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5GufDzOeFyQn",
        "colab_type": "text"
      },
      "source": [
        "## <font color=blue>Problem with Line Breaks</font>\n",
        "\n",
        "<div class=\"alert alert-info\" style=\"margin: 20px\">There's a <a href='https://github.com/explosion/spaCy/issues/1717'>known issue</a> with <strong>spaCy v2.0.12</strong> where some linebreaks are interpreted as `GPE` entities:</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hnrh9ipdFyQo",
        "colab_type": "code",
        "outputId": "607d4651-688d-4def-e116-9085691616d5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "spacy.__version__"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'2.1.4'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "902BVPS2FyQr",
        "colab_type": "code",
        "outputId": "bb95a8c0-219d-43ff-d6e1-4f0ec01093e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "doc = nlp(u'Originally priced at $29.50,\\nthe sweater was marked down to five dollars.')\n",
        "\n",
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "29.50 - MONEY - Monetary values, including unit\n",
            "five dollars - MONEY - Monetary values, including unit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHYmRCxYFyQt",
        "colab_type": "text"
      },
      "source": [
        "### <font color=blue>However, there is a simple fix that can be added to the nlp pipeline:</font>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PDSoRI5wFyQu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Quick function to remove ents formed on whitespace:\n",
        "def remove_whitespace_entities(doc):\n",
        "    doc.ents = [e for e in doc.ents if not e.text.isspace()]\n",
        "    return doc\n",
        "\n",
        "# Insert this into the pipeline AFTER the ner component:\n",
        "nlp.add_pipe(remove_whitespace_entities, after='ner')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ruQbOCoyFyQv",
        "colab_type": "code",
        "outputId": "b2fc4a24-b493-496a-a140-ced006e68d6c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# Rerun nlp on the text above, and show ents:\n",
        "doc = nlp(u'Originally priced at $29.50,\\nthe sweater was marked down to five dollars.')\n",
        "\n",
        "show_ents(doc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "29.50 - MONEY - Monetary values, including unit\n",
            "five dollars - MONEY - Monetary values, including unit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L_0o_qo5FyQx",
        "colab_type": "text"
      },
      "source": [
        "For more on **Named Entity Recognition** visit https://spacy.io/usage/linguistic-features#101"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rnuXo59CFyQx",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "## Noun Chunks\n",
        "`Doc.noun_chunks` are *base noun phrases*: token spans that include the noun and words describing the noun. Noun chunks cannot be nested, cannot overlap, and do not involve prepositional phrases or relative clauses.<br>\n",
        "Where `Doc.ents` rely on the **ner** pipeline component, `Doc.noun_chunks` are provided by the **parser**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5EBL_yb1FyQy",
        "colab_type": "text"
      },
      "source": [
        "### `noun_chunks` components:\n",
        "<table>\n",
        "<tr><td>`.text`</td><td>The original noun chunk text.</td></tr>\n",
        "<tr><td>`.root.text`</td><td>The original text of the word connecting the noun chunk to the rest of the parse.</td></tr>\n",
        "<tr><td>`.root.dep_`</td><td>Dependency relation connecting the root to its head.</td></tr>\n",
        "<tr><td>`.root.head.text`</td><td>The text of the root token's head.</td></tr>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JX28D_z9FyQy",
        "colab_type": "code",
        "outputId": "a3e6547a-4fb2-49dd-b2bb-885687ad221e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "doc = nlp(u\"Autonomous cars shift insurance liability toward manufacturers.\")\n",
        "\n",
        "for chunk in doc.noun_chunks:\n",
        "    print(chunk.text+' - '+chunk.root.text+' - '+chunk.root.dep_+' - '+chunk.root.head.text)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Autonomous cars - cars - nsubj - shift\n",
            "insurance liability - liability - dobj - shift\n",
            "manufacturers - manufacturers - pobj - toward\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "imJT3cP7FyQ1",
        "colab_type": "text"
      },
      "source": [
        "### `Doc.noun_chunks` is a  generator function\n",
        "Previously we mentioned that `Doc` objects do not retain a list of sentences, but they're available through the `Doc.sents` generator.<br>It's the same with `Doc.noun_chunks` - lists can be created if needed:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4e0yg8IUFyQ1",
        "colab_type": "code",
        "outputId": "08258aaa-4d85-452d-a012-a353f3629fa7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        }
      },
      "source": [
        "len(doc.noun_chunks)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-23-8b52b37c204e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdoc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnoun_chunks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m: object of type 'generator' has no len()"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ji9sjTVgFyQ3",
        "colab_type": "code",
        "outputId": "3f6a0053-4f0d-4726-8eeb-afcd553b512b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "len(list(doc.noun_chunks))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a_K9Cm3uFyQ6",
        "colab_type": "text"
      },
      "source": [
        "For more on **noun_chunks** visit https://spacy.io/usage/linguistic-features#noun-chunks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uOEPsXSPGcMC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SUfqIN7vIGJ-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Perform standard imports\n",
        "import spacy\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "\n",
        "# Import the displaCy library\n",
        "from spacy import displacy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gz-j8GpkIGKA",
        "colab_type": "code",
        "outputId": "e11a8944-2b76-4b88-a039-0ba949a86bce",
        "colab": {}
      },
      "source": [
        "doc = nlp(u'Over the last quarter Apple sold nearly 20 thousand iPods for a profit of $6 million. '\n",
        "         u'By contrast, Sony sold only 7 thousand Walkman music players.')\n",
        "\n",
        "displacy.render(doc, style='ent', jupyter=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">Over \n",
              "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    the last quarter\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    nearly 20 thousand\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    iPods\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " for a profit of \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    $6 million\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              ". By contrast, \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Sony\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    only 7 thousand\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Walkman\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " music players.</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XebKurgEIgDq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1hFosKW7IGKD",
        "colab_type": "code",
        "outputId": "41c61c67-414b-4403-b9f3-a93726a57760",
        "colab": {}
      },
      "source": [
        "for sent in doc.sents:\n",
        "    displacy.render(nlp(sent.text), style='ent', jupyter=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">Over \n",
              "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    the last quarter\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    nearly 20 thousand\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    iPods\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " for a profit of \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    $6 million\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              ".</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">By contrast, \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Sony\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    only 7 thousand\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Walkman\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " music players.</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v0uEcP0cIGKG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc2 = nlp(u'Over the last quarter Apple sold nearly 20 thousand iPods for a profit of $6 million. '\n",
        "         u'By contrast, my kids sold a lot of lemonade.')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NX_s73r2IGKI",
        "colab_type": "code",
        "outputId": "3acc985f-3bc9-4c91-d328-794b73692803",
        "colab": {}
      },
      "source": [
        "for sent in doc2.sents:\n",
        "    displacy.render(nlp(sent.text), style='ent', jupyter=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">Over \n",
              "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    the last quarter\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    nearly 20 thousand\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    iPods\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " for a profit of \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    $6 million\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              ".</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "C:\\Anaconda3\\lib\\runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n",
            "  \"__main__\", mod_spec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">By contrast, my kids sold a lot of lemonade.</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qXdOLILxIGKL",
        "colab_type": "code",
        "outputId": "9a29e302-6bcc-4081-ceef-8b4d2b5613a3",
        "colab": {}
      },
      "source": [
        "for sent in doc2.sents:\n",
        "    docx = nlp(sent.text)\n",
        "    if docx.ents:\n",
        "        displacy.render(docx, style='ent', jupyter=True)\n",
        "    else:\n",
        "        print(docx.text)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">Over \n",
              "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    the last quarter\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    nearly 20 thousand\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
              "</mark>\n",
              " \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    iPods\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " for a profit of \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    $6 million\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              ".</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "By contrast, my kids sold a lot of lemonade.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KyQvseRaIGKO",
        "colab_type": "code",
        "outputId": "9912c9cc-4a70-4520-c792-940c472cf936",
        "colab": {}
      },
      "source": [
        "options = {'ents': ['ORG', 'PRODUCT']}\n",
        "\n",
        "displacy.render(doc, style='ent', jupyter=True, options=options)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">Over the last quarter \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold nearly 20 thousand \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    iPods\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " for a profit of $6 million. By contrast, \n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Sony\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold only 7 thousand \n",
              "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Walkman\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " music players.</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dL7fp6miIGKQ",
        "colab_type": "code",
        "outputId": "b870e49a-1045-4689-e25c-abaf6dad8e38",
        "colab": {}
      },
      "source": [
        "colors = {'ORG': 'linear-gradient(90deg, #aa9cfc, #fc9ce7)', 'PRODUCT': 'radial-gradient(yellow, green)'}\n",
        "\n",
        "options = {'ents': ['ORG', 'PRODUCT'], 'colors':colors}\n",
        "\n",
        "displacy.render(doc, style='ent', jupyter=True, options=options)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div class=\"entities\" style=\"line-height: 2.5\">Over the last quarter \n",
              "<mark class=\"entity\" style=\"background: linear-gradient(90deg, #aa9cfc, #fc9ce7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold nearly 20 thousand \n",
              "<mark class=\"entity\" style=\"background: radial-gradient(yellow, green); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    iPods\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " for a profit of $6 million. By contrast, \n",
              "<mark class=\"entity\" style=\"background: linear-gradient(90deg, #aa9cfc, #fc9ce7); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Sony\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " sold only 7 thousand \n",
              "<mark class=\"entity\" style=\"background: radial-gradient(yellow, green); padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
              "    Walkman\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
              "</mark>\n",
              " music players.</div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cuZW1l1IGKT",
        "colab_type": "text"
      },
      "source": [
        "___\n",
        "# Creating Visualizations Outside of Jupyter\n",
        "If you're using another Python IDE or writing a script, you can choose to have spaCy serve up HTML separately.\n",
        "\n",
        "Instead of `displacy.render()`, use `displacy.serve()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C-xYfpzuIGKT",
        "colab_type": "code",
        "outputId": "c0226ed6-e700-4e77-84fe-7efa6ceabc70",
        "colab": {}
      },
      "source": [
        "displacy.serve(doc, style='ent', options=options)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "    Serving on port 5000...\n",
            "    Using the 'ent' visualizer\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "127.0.0.1 - - [04/Dec/2018 13:21:26] \"GET / HTTP/1.1\" 200 2210\n",
            "127.0.0.1 - - [04/Dec/2018 13:21:26] \"GET /favicon.ico HTTP/1.1\" 200 2210\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "    Shutting down server on port 5000.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}