Summarizing_Text_TextRank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization using the TextRank Algorithm\n",
    "* **Problem:** Automatically summarize long documents correctly to reduce reading time\n",
    "* **Solution:** Using graph-based method: TextRank algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Wikipedia url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://en.wikipedia.org/wiki/Natural_language_processing'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Web crawling using BeautifulSoup library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import urlopen\n",
    "\n",
    "# Fetch the data from Wikipedia\n",
    "page = urlopen(url)\n",
    "\n",
    "# Parse the data using BeautifulSoup\n",
    "soup = BeautifulSoup(page)\n",
    "\n",
    "# Get the text from the data\n",
    "text = ' '.join(map(lambda p: p.text, soup.find_all('p')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Convert text to string format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = str(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Summarize the text with ratio 0.1 (10% of the total words.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Since the so-called \"statistical revolution\"[15][16] in the late 1980s and mid-1990s, much natural language processing research has relied heavily on machine learning.\\nIncreasingly, however, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to each input feature (complex-valued embeddings,[17] and neural networks in general have also been proposed, for e.g. speech[18]).\\nHowever, part-of-speech tagging introduced the use of hidden Markov models to natural language processing, and increasingly, research has focused on statistical models, which make soft, probabilistic decisions based on attaching real-valued weights to the features making up the input data.\\nIn some areas, this shift has entailed substantial changes in how NLP systems are designed, such that deep neural network-based approaches may be viewed as a new paradigm distinct from statistical natural language processing.'"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.summarization.summarizer import summarize\n",
    "\n",
    "summarize(text, ratio=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'learn\\ncognitive\\ncognition\\nnlp\\nstatistical\\nrules\\ncomputers\\ncomputing\\ncomputational\\nmachine learning\\ntask\\ntasks\\nlarge\\nlargely\\ngrammar\\ngrammars\\nnatural language processing\\nlinguistics\\nprocess\\nprocesses\\nfeatures\\nfeature\\nreal\\nneural\\nmodeling\\nmodels\\nmodel\\nresults\\nsystems\\nresearch\\nresearched\\nanswers\\nanswering\\nhand\\nintelligence\\nintelligent'"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.summarization import keywords\n",
    "\n",
    "keywords(text, ratio=0.1)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "11938c6bc6919ae2720b4d5011047913343b08a43b18698fd82dedb0d4417594"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}