Spacy_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Spacy_Tutorial",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "192cyDm9db-p8iQlEJsvNoKYayumlm9Hx",
      "authorship_tag": "ABX9TyNAiTrD0A/eCCg5YtgacauW"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zciQ9FZG3DNe",
        "colab_type": "text"
      },
      "source": [
        "**What is SpaCy?**\n",
        "\n",
        "*   SpaCy is a free, open-source library for advanced NLP in Python.​\n",
        "*   It's written from the ground up in carefully memory-managed Cython.​\n",
        "*   It can be used to build information extraction or natural language    understanding systems, or to pre-process text for deep learning.\n",
        "\n",
        "\n",
        "**Features-**\n",
        "*   Non-destructive tokenization​\n",
        "*   Named entity recognition​\n",
        "*   Support for 59+ languages​\n",
        "*   46 statistical models for 16 languages​\n",
        "*   Pretrained word vectors​\n",
        "*   State-of-the-art speed​\n",
        "*   Easy deep learning integration\n",
        "*   Part-of-speech tagging\n",
        "*   Built in visualizers for syntax and NER"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CwEjBFeKBobA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import spacy"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OrNuyiTysZh4",
        "colab_type": "text"
      },
      "source": [
        "**Model Naming Convention-**\n",
        "\n",
        "[lang]\\_[name]\n",
        "\n",
        "[lang]\\_[type_genre_size]\n",
        "* **type:** Model capabilities (e.g. core for general-purpose model with vocabulary, syntax, entities and word vectors, or depent for only vocab, syntax and entities)\n",
        "* **genre:** Type of text the model is trained on (e.g. web for web text, news for news text)\n",
        "* **size:** Model size indicator (sm, md or lg)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bJt1KdW8wMBL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!pip install --user https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.0/en_core_web_md-2.2.0.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1rs-BRdnWHQ",
        "colab_type": "text"
      },
      "source": [
        "*Note-*\n",
        "\n",
        "Some models do not exist as a package in its own right on pypi.org or Anaconda, so you can't just pip install it by name. However you can find download links for the model on the [Github page](https://github.com/explosion/spacy-models) and you can pip install directly from one of the download URLs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_p0mLnQJKsJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "846baa80-010f-4c5b-f009-cf238f276895"
      },
      "source": [
        "#Download model --------->  python -m spacy download en_core_web_sm\n",
        "model = \"en_core_web_md\"\n",
        "nlp = spacy.load(model)\n",
        "type(nlp)\n",
        "#contains the processing pipeline\n",
        "#includes language-specific rules for tokenization etc."
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "spacy.lang.en.English"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VIeSW6zfwBx6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "c8c104d4-222a-4ef3-d1bc-76a6964f13bb"
      },
      "source": [
        "nlp.pipe_names"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['tagger', 'parser', 'ner']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKH7PVPl57FC",
        "colab_type": "text"
      },
      "source": [
        "![image](https://spacy.io/pipeline-7a14d4edd18f3edfee8f34393bff2992.svg)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nLBUaUrqJYCV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "text = '''Abraham Lincoln was an American statesman and lawyer who served as the 16th president of the United States.\n",
        "       Lincoln led the nation through its greatest moral, constitutional, and political crisis in the American Civil War'''"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rLpyXyYiKpqd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "doc = nlp(text)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SLaBEL5LMnrm",
        "colab_type": "text"
      },
      "source": [
        "*   The Doc behaves like a normal Python sequence and lets you iterate over its tokens, or get a token by its index.\n",
        "*   Even though a Doc is processed, it still holds all information of the original text, like whitespace characters. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F7OdAe6Xgol2",
        "colab_type": "text"
      },
      "source": [
        "**Tokenization**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Yu-V-ctKu0B",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "921b2f48-4d54-4c4e-9d98-a5e0a2ab784b"
      },
      "source": [
        "tokens = []\n",
        "for token in doc:\n",
        "  tokens.append(token.text)\n",
        "print(tokens)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['Abraham', 'Lincoln', 'was', 'an', 'American', 'statesman', 'and', 'lawyer', 'who', 'served', 'as', 'the', '16th', 'president', 'of', 'the', 'United', 'States', '.', '\\n       ', 'Lincoln', 'led', 'the', 'nation', 'through', 'its', 'greatest', 'moral', ',', 'constitutional', ',', 'and', 'political', 'crisis', 'in', 'the', 'American', 'Civil', 'War']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nP_nKx04O9QO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "2a4dbb68-bb65-45f6-e535-0eb54fde6883"
      },
      "source": [
        "#span\n",
        "span = doc[4:8]\n",
        "span.text"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic": {
              "type": "string"
            },
            "text/plain": [
              "'American statesman and lawyer'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CM6D3d7Ng9OL",
        "colab_type": "text"
      },
      "source": [
        "**Punctuation and Stop Words**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v7dk31eSeg36",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 426
        },
        "outputId": "e39fb7db-4733-4234-94d7-d0285e7f46e7"
      },
      "source": [
        "from spacy.lang.en.stop_words import STOP_WORDS as stopwords\n",
        "print(len(stopwords))\n",
        "pp_text = [token.text for token in doc if not (token.is_punct or token.is_stop)]\n",
        "pp_text"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "326\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Abraham',\n",
              " 'Lincoln',\n",
              " 'American',\n",
              " 'statesman',\n",
              " 'lawyer',\n",
              " 'served',\n",
              " '16th',\n",
              " 'president',\n",
              " 'United',\n",
              " 'States',\n",
              " '\\n       ',\n",
              " 'Lincoln',\n",
              " 'led',\n",
              " 'nation',\n",
              " 'greatest',\n",
              " 'moral',\n",
              " 'constitutional',\n",
              " 'political',\n",
              " 'crisis',\n",
              " 'American',\n",
              " 'Civil',\n",
              " 'War']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9GBvQRQhXsO",
        "colab_type": "text"
      },
      "source": [
        "**Lemmatization, POS, NER and Dependency Parsing**\n",
        "\n",
        "*   .pos_ - Course grained parts of speech\n",
        "*   .tag_ - Fine grained parts of speech\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wlOD5tOttVf5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 728
        },
        "outputId": "7760cc03-c951-464c-bbcf-819743b7ee73"
      },
      "source": [
        "import pandas as pd\n",
        "info = []\n",
        "for token in doc:\n",
        "  info.append([token , token.lemma_ , token.ent_type_ , token.pos_ , token.tag_ , token.dep_ ])\n",
        "info = pd.DataFrame(info ,columns= ['token' , 'lemma' , 'ent' ,'pos' , 'tag' , 'dep'])\n",
        "print(info)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "             token           lemma      ent    pos   tag       dep\n",
            "0          Abraham         Abraham   PERSON  PROPN   NNP  compound\n",
            "1          Lincoln         Lincoln   PERSON  PROPN   NNP     nsubj\n",
            "2              was              be             AUX   VBD      ROOT\n",
            "3               an              an             DET    DT       det\n",
            "4         American        american     NORP    ADJ    JJ      amod\n",
            "5        statesman       statesman            NOUN    NN      attr\n",
            "6              and             and           CCONJ    CC        cc\n",
            "7           lawyer          lawyer            NOUN    NN      conj\n",
            "8              who             who            PRON    WP     nsubj\n",
            "9           served           serve            VERB   VBD     relcl\n",
            "10              as              as           SCONJ    IN      prep\n",
            "11             the             the             DET    DT       det\n",
            "12            16th            16th  ORDINAL    ADJ    JJ      amod\n",
            "13       president       president            NOUN    NN      pobj\n",
            "14              of              of             ADP    IN      prep\n",
            "15             the             the      GPE    DET    DT       det\n",
            "16          United          United      GPE  PROPN   NNP  compound\n",
            "17          States          States      GPE  PROPN   NNP      pobj\n",
            "18               .               .           PUNCT     .     punct\n",
            "19       \\n              \\n                  SPACE   _SP          \n",
            "20         Lincoln         Lincoln      ORG  PROPN   NNP     nsubj\n",
            "21             led            lead            VERB   VBD      ROOT\n",
            "22             the             the             DET    DT       det\n",
            "23          nation          nation            NOUN    NN      dobj\n",
            "24         through         through             ADP    IN      prep\n",
            "25             its          -PRON-            PRON  PRP$      poss\n",
            "26        greatest           great             ADJ   JJS      amod\n",
            "27           moral           moral             ADJ    JJ      amod\n",
            "28               ,               ,           PUNCT     ,     punct\n",
            "29  constitutional  constitutional             ADJ    JJ      conj\n",
            "30               ,               ,           PUNCT     ,     punct\n",
            "31             and             and           CCONJ    CC        cc\n",
            "32       political       political             ADJ    JJ      conj\n",
            "33          crisis          crisis            NOUN    NN      pobj\n",
            "34              in              in             ADP    IN      prep\n",
            "35             the             the    EVENT    DET    DT       det\n",
            "36        American        American    EVENT  PROPN   NNP  compound\n",
            "37           Civil           Civil    EVENT  PROPN   NNP  compound\n",
            "38             War             War    EVENT  PROPN   NNP      pobj\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AO56w2e9loRN",
        "colab_type": "text"
      },
      "source": [
        "https://universaldependencies.org/docs/u/pos/\n",
        "\n",
        "* ADJ: adjective\n",
        "* ADP: adposition\n",
        "* ADV: adverb\n",
        "* AUX: auxiliary verb\n",
        "* CONJ: coordinating conjunction\n",
        "* DET: determiner\n",
        "* INTJ: interjection\n",
        "* NOUN: noun\n",
        "* NUM: numeral\n",
        "* PART: particle\n",
        "* PRON: pronoun\n",
        "* PROPN: proper noun\n",
        "* PUNCT: punctuation\n",
        "* SCONJ: subordinating conjunction\n",
        "* SYM: symbol\n",
        "* VERB: verb\n",
        "* X: other"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhZ0KSU8jC6z",
        "colab_type": "text"
      },
      "source": [
        "**DisplaCy**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ciuvP9Mx1Qi9",
        "colab_type": "text"
      },
      "source": [
        "The dependency visualizer, dep, shows part-of-speech tags and syntactic dependencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "swXQfdFaV9mm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 292
        },
        "outputId": "639c7966-10de-4ac1-f62c-47daafc1b00d"
      },
      "source": [
        "from spacy import displacy\n",
        "doc2 = nlp(\"Apple is looking at buying U.K. startup for $1 billion\")\n",
        "displacy.render(doc2 ,style = 'dep' ,jupyter=True , options= {'distance': 90 , 'compact' :False})\n",
        "#https://spacy.io/usage/visualizers"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"4bf1e96f8f80465392402bcf2d2a8177-0\" class=\"displacy\" width=\"1040\" height=\"272.0\" direction=\"ltr\" style=\"max-width: none; height: 272.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Apple</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"140\">is</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"140\">AUX</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"230\">looking</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"230\">VERB</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"320\">at</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"320\">ADP</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"410\">buying</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"410\">VERB</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"500\">U.K.</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"500\">PROPN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"590\">startup</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"590\">NOUN</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"680\">for</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"680\">ADP</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"770\">$</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"770\">SYM</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"860\">1</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"860\">NUM</tspan>\n",
              "</text>\n",
              "\n",
              "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"182.0\">\n",
              "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"950\">billion</tspan>\n",
              "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"950\">NUM</tspan>\n",
              "</text>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-0\" stroke-width=\"2px\" d=\"M70,137.0 C70,47.0 225.0,47.0 225.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M70,139.0 L62,127.0 78,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-1\" stroke-width=\"2px\" d=\"M160,137.0 C160,92.0 220.0,92.0 220.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M160,139.0 L152,127.0 168,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-2\" stroke-width=\"2px\" d=\"M250,137.0 C250,92.0 310.0,92.0 310.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M310.0,139.0 L318.0,127.0 302.0,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-3\" stroke-width=\"2px\" d=\"M340,137.0 C340,92.0 400.0,92.0 400.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pcomp</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M400.0,139.0 L408.0,127.0 392.0,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-4\" stroke-width=\"2px\" d=\"M520,137.0 C520,92.0 580.0,92.0 580.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M520,139.0 L512,127.0 528,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-5\" stroke-width=\"2px\" d=\"M430,137.0 C430,47.0 585.0,47.0 585.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M585.0,139.0 L593.0,127.0 577.0,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-6\" stroke-width=\"2px\" d=\"M430,137.0 C430,2.0 680.0,2.0 680.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M680.0,139.0 L688.0,127.0 672.0,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-7\" stroke-width=\"2px\" d=\"M790,137.0 C790,47.0 945.0,47.0 945.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-7\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">quantmod</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M790,139.0 L782,127.0 798,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-8\" stroke-width=\"2px\" d=\"M880,137.0 C880,92.0 940.0,92.0 940.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-8\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M880,139.0 L872,127.0 888,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "\n",
              "<g class=\"displacy-arrow\">\n",
              "    <path class=\"displacy-arc\" id=\"arrow-4bf1e96f8f80465392402bcf2d2a8177-0-9\" stroke-width=\"2px\" d=\"M700,137.0 C700,2.0 950.0,2.0 950.0,137.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
              "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
              "        <textPath xlink:href=\"#arrow-4bf1e96f8f80465392402bcf2d2a8177-0-9\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
              "    </text>\n",
              "    <path class=\"displacy-arrowhead\" d=\"M950.0,139.0 L958.0,127.0 942.0,127.0\" fill=\"currentColor\"/>\n",
              "</g>\n",
              "</svg></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rDRGPGfAYxpL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "640ba933-3953-4865-e501-b6cbf44a1732"
      },
      "source": [
        "displacy.render(doc2 ,style = 'ent'  ,jupyter = True, options= {'distance': 90 , 'compact' :True})"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
              "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    Apple\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
              "</mark>\n",
              " is looking at buying \n",
              "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    U.K.\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
              "</mark>\n",
              " startup for \n",
              "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
              "    $1 billion\n",
              "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
              "</mark>\n",
              "</div></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7p9Q675Htg1Z",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b8623838-c4d7-4c5d-e4bf-411c61308697"
      },
      "source": [
        "print(doc[1].ent_type_ , spacy.explain(doc[1].ent_type_))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "PERSON People, including fictional\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vuxgckW9HkxD",
        "colab_type": "text"
      },
      "source": [
        "**Updating pre-trained SpaCy model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GY147FezPC54",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LABEL = \"PERSON\"\n",
        "train_data = [\n",
        "        (\"Uber blew through $1 million a week\", {\"entities\": [(0, 4, \"ORG\")]}),\n",
        "        (\"Google rebrands its business apps\", {\"entities\": [(0, 6, \"ORG\")]}),\n",
        "        (\"Barak Obama is an American politician and attorney who served as the 44th president of the U.S.\", {\"entities\": [(0, 12, \"PERSON\")]}),\n",
        "        (\"Obama was the first African-American president of the U.S.\", {\"entities\": [(0, 5, \"PERSON\")]})]"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cezicrvcA3UB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 550
        },
        "outputId": "d9b24d52-0e34-4bd8-a226-6730195e859e"
      },
      "source": [
        "#Adding a new entity type to NER model\n",
        "from spacy.util import minibatch, compounding\n",
        "import warnings\n",
        "import random\n",
        "\n",
        "ner = nlp.get_pipe(\"ner\")\n",
        "ner.add_label(LABEL)\n",
        "optimizer = nlp.begin_training()\n",
        "\n",
        "# get names of other pipes to disable them during training\n",
        "pipe_exceptions = [\"ner\", \"trf_wordpiecer\", \"trf_tok2vec\"]\n",
        "other_pipes = [pipe for pipe in nlp.pipe_names if pipe not in pipe_exceptions]\n",
        "\n",
        "with nlp.disable_pipes(*other_pipes):\n",
        "  warnings.filterwarnings(\"once\", category=UserWarning, module='spacy')\n",
        "  # batch up the examples using spaCy's minibatch\n",
        "  for itn in range(30):\n",
        "    random.shuffle(train_data)\n",
        "    batches = minibatch(train_data, size=compounding(1.0, 4.0, 1.001))\n",
        "    losses = {}\n",
        "    for batch in batches:\n",
        "      texts, annotations = zip(*batch)\n",
        "      nlp.update(texts, annotations, sgd=optimizer, drop=0.2, losses=losses)\n",
        "    print(\"Losses\", losses)\n",
        "\n"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Losses {'ner': 23.273035168647766}\n",
            "Losses {'ner': 19.016400575637817}\n",
            "Losses {'ner': 15.277870684862137}\n",
            "Losses {'ner': 10.793894559144974}\n",
            "Losses {'ner': 6.7386059165000916}\n",
            "Losses {'ner': 6.889925971627235}\n",
            "Losses {'ner': 13.292466878890991}\n",
            "Losses {'ner': 5.45738297700882}\n",
            "Losses {'ner': 5.399205908179283}\n",
            "Losses {'ner': 6.555235326290131}\n",
            "Losses {'ner': 2.365303799510002}\n",
            "Losses {'ner': 2.284915106371045}\n",
            "Losses {'ner': 2.3383300537243485}\n",
            "Losses {'ner': 2.858860532986}\n",
            "Losses {'ner': 2.938103277236223}\n",
            "Losses {'ner': 3.835870539303869}\n",
            "Losses {'ner': 1.374498747754842}\n",
            "Losses {'ner': 1.7992042694240808}\n",
            "Losses {'ner': 1.5569051960483193}\n",
            "Losses {'ner': 0.4611848518252373}\n",
            "Losses {'ner': 2.0489995032548904}\n",
            "Losses {'ner': 0.18306645564734936}\n",
            "Losses {'ner': 1.8202454571146518}\n",
            "Losses {'ner': 0.9124089432880282}\n",
            "Losses {'ner': 0.43469150067539886}\n",
            "Losses {'ner': 1.6597606280702166}\n",
            "Losses {'ner': 0.9416989590390585}\n",
            "Losses {'ner': 0.30820509951445274}\n",
            "Losses {'ner': 0.10511063120793551}\n",
            "Losses {'ner': 0.10097842956565728}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r_C9LCJ3EYUn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "ece8a62d-3ba6-4eae-cbfa-e09b98319742"
      },
      "source": [
        "# test the trained model\n",
        "test_text = \"Barak served as 16th U.S. president.\"\n",
        "test_doc = nlp(test_text)\n",
        "print(\"Entities in '%s'\" % test_text)\n",
        "for ent in test_doc.ents:\n",
        "  print(ent.label_, ent.text)\n"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Entities in 'Barak served as 16th U.S. president.'\n",
            "PERSON Barak\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PYGMQD0njU8d",
        "colab_type": "text"
      },
      "source": [
        "**Noun Phrase Detection**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fzH4nng_w3zn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "3ab9e8f2-2b86-4364-cda6-3ac2ae186634"
      },
      "source": [
        "# Extract Noun Phrases\n",
        "for chunk in doc.noun_chunks:\n",
        "  print(chunk)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Abraham Lincoln\n",
            "an American statesman\n",
            "lawyer\n",
            "who\n",
            "the 16th president\n",
            "the United States\n",
            "Lincoln\n",
            "the nation\n",
            "its greatest moral, constitutional, and political crisis\n",
            "the American Civil War\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJOXgvUpJ6oj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 390
        },
        "outputId": "364272ed-de37-46c5-91a6-0e0f21d16a7e"
      },
      "source": [
        "#Retokenization\n",
        "with doc.retokenize() as retokenizer:\n",
        "  for chunks in doc.noun_chunks:\n",
        "      retokenizer.merge(chunks)\n",
        "\n",
        "for token in doc:\n",
        "  print(token)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Abraham Lincoln\n",
            "was\n",
            "an American statesman\n",
            "and\n",
            "lawyer\n",
            "who\n",
            "served\n",
            "as\n",
            "the 16th president\n",
            "of\n",
            "the United States\n",
            ".\n",
            "\n",
            "       \n",
            "Lincoln\n",
            "led\n",
            "the nation\n",
            "through\n",
            "its greatest moral, constitutional, and political crisis\n",
            "in\n",
            "the American Civil War\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68A0NLjekTMU",
        "colab_type": "text"
      },
      "source": [
        "**Rule-Based Matching**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2eIKqXm52gvK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "53c0b166-da63-4f1e-fad9-38dc29ae676a"
      },
      "source": [
        "new_doc = nlp(text)\n",
        "from spacy.matcher import Matcher\n",
        "matcher = Matcher(nlp.vocab)\n",
        "def extract_full_name(doc):\n",
        "  pattern = [{'POS': 'PROPN'}, {'POS': 'PROPN'}]\n",
        "  matcher.add('FULL_NAME', None, pattern)\n",
        "  matches = matcher(doc)\n",
        "  for match_id, start, end in matches:\n",
        "    span = doc[start:end]\n",
        "    print(span.text)\n",
        "  return\n",
        "\n",
        "extract_full_name(new_doc)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Abraham Lincoln\n",
            "United States\n",
            "American Civil\n",
            "Civil War\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TTV_XcfJjkO4",
        "colab_type": "text"
      },
      "source": [
        "**Custom Pipeline Component**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_BusThO8sz6G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def set_custom_boundaries(doc):\n",
        "  # Adds support to use `, ` as the delimiter for sentence detection\n",
        "  for token in doc[:-1]:\n",
        "    if token.text == ',':\n",
        "      doc[token.i+1].is_sent_start = True\n",
        "  return doc"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nw7X1uYY30Lu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4c306e74-acf4-4f92-f5b8-6ce09a6caf18"
      },
      "source": [
        "c_text = '''The fox jumps over the dog, the fox is very clever and quick. The dog is slow and lazy.The cat is smarter than the fox and the dog.'''\n",
        "custom_nlp = spacy.load(model)\n",
        "custom_nlp.add_pipe(set_custom_boundaries, before='parser')\n",
        "c_doc = custom_nlp(c_text)\n",
        "print(custom_nlp.pipe_names)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['tagger', 'set_custom_boundaries', 'parser', 'ner']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ydO1ghKzzrCg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "9c139b74-f508-4633-94f2-16db506fdc73"
      },
      "source": [
        "# Sentence Detection with customization\n",
        "sentences = list(c_doc.sents)\n",
        "for idx ,sentence in zip(range(len(sentences)) , sentences):\n",
        "  print(idx ,sentence)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0 The fox jumps over the dog,\n",
            "1 the fox is very clever and quick.\n",
            "2 The dog is slow and lazy.\n",
            "3 The cat is smarter than the fox and the dog.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDzMZr-EaOD8",
        "colab_type": "text"
      },
      "source": [
        "**Word Vectors**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v32oQdKUamHU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 906
        },
        "outputId": "99c31405-858b-4375-86d2-e3e8c33dbe4a"
      },
      "source": [
        "monkey_doc = nlp.vocab['monkey']\n",
        "print(monkey_doc.vector)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.016269  -0.66774   -0.21387   -0.47919   -0.62437    0.55147\n",
            " -0.091257  -0.26658    0.36018    0.88362   -0.65668   -0.6079\n",
            " -0.23139   -0.03844    0.055579   0.30017   -0.040855   0.42249\n",
            "  0.16564   -0.32602   -0.015816  -0.1382     0.031401  -0.044665\n",
            "  0.036645  -0.26125   -0.15189   -0.32714    0.28742   -0.13792\n",
            " -0.20004   -0.083271   0.50561    0.11606   -0.084337  -0.05022\n",
            "  0.11342    0.060052  -0.39589   -0.20859    0.35132    0.044604\n",
            " -0.48136    0.18762    0.14262    0.12414   -0.039411  -0.035831\n",
            "  0.18882    0.2482     0.014472  -0.101      0.034416  -0.09526\n",
            " -0.10771    0.14274    0.44874    0.16699    0.33336   -0.15883\n",
            " -0.29088    0.3315    -0.014055   0.11276   -0.1387    -0.47695\n",
            " -0.26999   -0.047984   0.10394    0.010292  -0.26874   -0.23449\n",
            " -0.25156    0.046208  -0.041264  -0.048494  -0.39944   -0.26964\n",
            "  0.087368  -0.19203   -0.24358    0.32748   -0.10197   -0.14169\n",
            " -0.67555   -0.2514     1.0979     0.304     -0.1469    -0.1116\n",
            "  0.4606     0.30304   -0.40118   -0.2089    -0.30873   -0.33242\n",
            " -0.72157   -0.47915    0.25077    0.085148   0.16692    0.18554\n",
            " -0.29145    0.11434    0.19817   -0.73486   -0.45073    0.2591\n",
            "  0.57499    0.51814    0.18171    0.12665   -0.027975  -0.1428\n",
            "  0.35926    0.10777    0.27618   -0.065737   0.022499   0.24596\n",
            "  0.20999   -0.55237    0.36114   -0.61658   -0.95942    0.04245\n",
            " -0.42379    0.32597   -0.96621    0.31709    0.052349  -0.28754\n",
            " -0.41789    0.016178  -0.14216   -0.36823    0.097384   0.30942\n",
            " -0.34188    0.63268   -3.0487     0.41506   -0.42276   -0.29162\n",
            "  0.13273    0.090978   0.50537    0.33569    0.15649   -0.61364\n",
            " -0.078473   0.052903  -0.089202  -0.17006   -0.10629    0.072781\n",
            " -0.18208   -0.12121   -0.12316   -0.2577     0.49855    0.38823\n",
            " -0.34713   -0.022198  -0.057936  -0.36363   -0.42063   -0.15234\n",
            " -0.11713   -0.0057556  0.28293    0.26144    0.17048   -0.069129\n",
            " -0.57854    0.077801  -0.15415    0.15217    0.064587   0.072366\n",
            "  0.11501    0.16923    0.18542   -0.11612    0.17716    0.065526\n",
            "  0.056415   0.0063093  0.17094   -0.22847   -0.41313   -0.18356\n",
            " -0.28608    0.25528   -0.27836   -0.12389    0.10491    0.13823\n",
            "  0.19715   -0.30345   -0.52606    0.36995   -0.10243    0.082798\n",
            "  0.45138   -0.47898    0.42201   -0.26057    0.176     -0.40755\n",
            "  0.34136    0.37939    0.048573   0.31388    0.17125   -0.26084\n",
            " -0.16905    0.21174   -0.19714   -0.42361    0.039382   0.024057\n",
            "  0.0055536 -0.97202   -0.13959    0.18079    0.24073    0.66468\n",
            "  0.19451    0.5476    -0.3345     0.37887   -0.39857   -0.11478\n",
            "  0.004402  -0.13836   -0.34034    0.28988    0.48544    0.0090168\n",
            " -0.29443   -0.52597   -0.0097584 -0.20527    0.10202   -0.322\n",
            "  0.097427  -0.33928   -0.51096    0.32621    0.12059    0.046824\n",
            "  0.11217    0.33181    0.6395     0.08705   -0.35301    0.13231\n",
            "  0.4115    -0.19986   -0.35215    0.077475  -0.16267   -0.26076\n",
            " -0.16936    0.45028   -0.25797    0.25028    0.50296    0.34695\n",
            " -0.6011     0.067547  -0.013422   0.1323     0.14097   -0.085268\n",
            "  0.40054    0.1654     0.065703   0.047816   0.040342   0.38655\n",
            "  0.026661   0.16152    0.090193   0.16698    0.070179  -0.11018\n",
            " -0.10715   -0.15106    0.0054558  0.087071  -0.31702    0.49047\n",
            " -0.24666    0.65551    0.14095    0.53461   -0.32408    0.24846  ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oquRATZGa90B",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 248
        },
        "outputId": "57d00ba6-34d6-409a-96c6-5aea96a15a70"
      },
      "source": [
        "unknown_doc = nlp('dgdgs')\n",
        "print(unknown_doc.vector)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TE6fH6F9a-wa",
        "colab_type": "text"
      },
      "source": [
        "**Custom word vectors**\n",
        "\n",
        "*   Custom word vectors can be trained using a number of open-source libraries, such as Gensim, Fast Text, or Tomas Mikolov’s original word2vec implementation.\n",
        "\n",
        "*  For everyday use, we want to convert the vectors model into a binary format that loads faster and takes up less space on disk. The easiest way to do this is the [init-model](https://spacy.io/api/cli#init-model) command-line utility.\n",
        "\n",
        "python -m spacy init-model [lang] [output_dir]\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aHD2rIUvcoXS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Example-\n",
        "#wget https://s3-us-west-1.amazonaws.com/fasttext-vectors/word-vectors-v2/cc.la.300.vec.gz\n",
        "#python -m spacy init-model en /tmp/la_vectors_wiki_lg --vectors-loc cc.la.300.vec.gz\n",
        "#nlp_latin = spacy.load(\"/tmp/la_vectors_wiki_lg\")"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fATuHa8EjtR7",
        "colab_type": "text"
      },
      "source": [
        "**Similarity**\n",
        "*   By default spaCy calculates cosine similarity\n",
        "*   Each Doc, Span and Token comes with a [.similarity()](https://spacy.io/api/token#similarity) method that lets you compare it with another object, and determine the similarity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4tTLlUe_6IHi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "23a82668-f289-4c69-d301-5134eb97355d"
      },
      "source": [
        "banana = nlp.vocab['banana']\n",
        "dog = nlp.vocab['dog']\n",
        "fruit = nlp.vocab['fruit']\n",
        "animal = nlp.vocab['animal']\n",
        "\n",
        "print('dog and banana', dog.similarity(banana))\n",
        "print('dog and animal', dog.similarity(animal))\n",
        "print('dog and fruit ', dog.similarity(fruit))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dog and banana 0.24327643\n",
            "dog and animal 0.66185343\n",
            "dog and fruit  0.23552851\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IY--jj7RkFxM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "934c42f4-3905-4bca-c8a8-cd5beccf94e4"
      },
      "source": [
        "text1 = 'How can I end violence?'\n",
        "text2 = 'What should I do to be a peaceful?'\n",
        "doc1 = nlp(text1)\n",
        "doc2 = nlp(text2)\n",
        "print(\"Similarity :\", doc1.similarity(doc2))\n"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Similarity : 0.9165530875252746\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gM8b8T7m4WHZ",
        "colab_type": "text"
      },
      "source": [
        "**References-**\n",
        "\n",
        "https://course.spacy.io/en\n",
        "\n",
        "https://spacy.io/usage\n",
        "\n",
        "https://realpython.com/natural-language-processing-spacy-python/\n",
        "\n"
      ]
    }
  ]
}