push-hugginface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i3uVsBf1jacE"
   },
   "source": [
    "# AML Project - Abstractive Text Summarization\n",
    "\n",
    "## Code for pushing trained model to huggingface\n",
    "\n",
    "Problem Statment: Given a news article, generate a summary of two-to-three sentences and a headline for the article.\n",
    "\n",
    "The summary should be abstractive rather than extractive. In abstractive summarization, new sentences are generated as part of the summary and the sentences in the summary might not be present in the news article."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "quywxuE8jxy9"
   },
   "source": [
    "### Importing necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 74,
     "output_embedded_package_id": "1TA7HShfr2cnAW-d--z5RbXOqzAKiywKT"
    },
    "executionInfo": {
     "elapsed": 4140,
     "status": "ok",
     "timestamp": 1652119490916,
     "user": {
      "displayName": "Kuber Shahi",
      "userId": "04157404054756166055"
     },
     "user_tz": -330
    },
    "id": "2ghsLfkLj0KB",
    "outputId": "240bede0-e65c-4a1f-e9b7-3da1a4b26fb5"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# import pegasus generator and tokenizer\n",
    "from transformers import PegasusForConditionalGeneration, PegasusTokenizerFast\n",
    "from huggingface_hub import whoami, HfFolder, create_repo\n",
    "\n",
    "# visualisation\n",
    "# import plotly.express as px\n",
    "# import plotly.graph_objects as go\n",
    "# import plotly.offline as pyo\n",
    "\n",
    "# pyo.init_notebook_mode() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "609e90fdc4cb4b15899be97a5d56b9c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for logging to huggingface\n",
    "from huggingface_hub import notebook_login\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.1\n",
      "cpu\n"
     ]
    }
   ],
   "source": [
    "print(torch.__version__)\n",
    "torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "print(torch_device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of PegasusForConditionalGeneration were not initialized from the model checkpoint at google/pegasus-large and are newly initialized: ['model.encoder.embed_positions.weight', 'model.decoder.embed_positions.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model_path = \"../../E/GH/model/checkpoint-47000\"\n",
    "pegasus_tokenizer = PegasusTokenizerFast.from_pretrained(\"google/pegasus-large\")\n",
    "pegasus_large = PegasusForConditionalGeneration.from_pretrained(\"google/pegasus-large\").to(torch_device)\n",
    "pegasus_finetuned = PegasusForConditionalGeneration.from_pretrained(model_path, local_files_only=True).to(torch_device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the model on a test case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to get summary of a text of list of texts\n",
    "def get_summary(tokenizer, model, x):\n",
    "    x_tokenized = tokenizer(x, truncation=True, padding = True, return_tensors=\"pt\").to(torch_device)\n",
    "    print(\"Input X tokenized. Generating Summary ...\")\n",
    "    y_pred_tokenized= model.generate(**x_tokenized).to(torch_device)\n",
    "    print(\"Summary Generated. Decoding Summary ...\")\n",
    "    y_pred = tokenizer.batch_decode(y_pred_tokenized, skip_special_tokens=True)\n",
    "    print(\"Summary Decoded.\")\n",
    "    return y_pred\n",
    "\n",
    "# function to caluculate rogue score\n",
    "def calculate_rouge(m, y_pred, y):\n",
    "    candidate = [i.split() for i in y_pred]\n",
    "    reference = [i.split() for i in y]\n",
    "    # print(candidate, reference)\n",
    "    \n",
    "    m.update((candidate, reference))\n",
    "    \n",
    "    return m.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_text = \"\"\"\n",
    "Elon Musk said on Tuesday that he would “reverse the permanent ban” of former President Donald J. Trump on Twitter and let him back on the social network, in one of the first specific comments by Mr. Musk, the world’s richest man, of how he would change the social media service.\n",
    "\n",
    "Mr. Musk, who struck a deal last month to buy Twitter for $44 billion, said at a Financial Times conference that the company’s decision to bar Mr. Trump last year for tweets about the riots at the U.S. Capitol was “a mistake because it alienated a large part of the country and did not ultimately result in Donald Trump not having a voice.” He added that it was “morally wrong and flat-out stupid” and that “permanent bans just fundamentally undermine trust in Twitter.”\n",
    "\n",
    "Mr. Musk’s remarks were a preview of the kinds of sweeping changes he might make at Twitter, which he is expected to take ownership of in the next six months. The billionaire, who also leads the electric carmaker Tesla and the rocket company SpaceX, has called himself a “free speech absolutist” and has said he is unhappy with how Twitter decides what can and cannot be posted online.\n",
    "\n",
    "But up until Tuesday, Mr. Musk, 50, had spoken mostly in general terms and had not singled out Twitter accounts that might be affected by his takeover. He had called free speech “the bedrock of a functioning democracy” and had spoken of his desire to give people more control over their own social media feeds. By specifying that Mr. Trump could return to the platform, Mr. Musk uncorked a political firestorm.\n",
    "\n",
    "Mr. Trump wielded Twitter for many years as both a megaphone and a cudgel, rallying his 88 million followers on issues such as immigration while also going after opponents. That avenue was cut off in January 2021 when Twitter, along with Facebook and other platforms, barred Mr. Trump from posting in the wake of the attack on the U.S. Capitol building. Twitter said at the time that Mr. Trump had violated policies and risked inciting violence among his supporters. Facebook barred Mr. Trump for similar reasons.\n",
    "\n",
    "Mr. Trump, who has since begun a social media site called Truth Social, did not respond to a request for comment. Last month, Mr. Trump said that even with Mr. Musk buying Twitter, he did not plan to return to the platform and was “going to stay on Truth.”\n",
    "\n",
    "Twitter declined to comment.\n",
    "\n",
    "Derrick Johnson, the president of the N.A.A.C.P., said that free speech online needed to come with guardrails.\n",
    "\n",
    "“Mr. Musk: Free speech is wonderful, hate speech is unacceptable,” he said. “Do not allow 45 to return to the platform. Do not allow Twitter to become a petri dish for hate speech or falsehoods that subvert our democracy.”\n",
    "\n",
    "But Jack Dorsey, a founder and board member of Twitter, tweeted that permanent suspensions of individual users “are a failure” of the company and largely “don’t work.” Mr. Dorsey, who was chief executive of Twitter when Mr. Trump was barred, had said last year that booting the president was the right decision for the company, but backtracked on Tuesday by calling it “a business decision” and saying “it shouldn’t have been.”\n",
    "\n",
    "Even with Mr. Musk’s comments, Mr. Trump’s potential return to Twitter remains far from assured. Mr. Musk is mercurial and has a history of saying things he does not follow through on. In 2018, he famously declared that he planned to take Tesla private and had the funding to do so, when he did not. Even his most devoted followers sometimes wonder whether his obscurantist tweets are serious or are made in jest.\n",
    "\n",
    "Investors have also questioned whether Mr. Musk’s deal for Twitter will be completed. The company’s shares closed Tuesday at $47.26, well below the $54.20 that Mr. Musk agreed to pay for them. Mr. Musk is also still securing financing for his takeover. While venture-capital firms and some big banks have lined up to invest, the billionaire is on the hook to provide as much as $21 billion of his own cash. He has not detailed where he will obtain that money.\n",
    "\n",
    "Mr. Musk referred to the possibility of the deal not closing on Tuesday and of nothing happening with Mr. Trump’s Twitter account. “Obviously, I don’t own Twitter yet,” he said at the Financial Times conference. “So this is not a thing that will definitely happen because what if I don’t own Twitter.”\n",
    "\n",
    "Mr. Trump long bedeviled social media companies because of how he pushed the line on speech, sometimes spreading lies and bullying people. But the companies’ moves to bar him drew accusations, especially from conservatives, that they were engaging in censorship and were biased against Republican voices.\n",
    "\n",
    "Mr. Musk seemed to echo some of those conservative complaints on Tuesday, accusing Twitter of “a strong left bias, because it’s based in San Francisco” and saying “victory would be that the most far-right 10 percent and the most far-left 10 percent are equally upset.”\n",
    "\n",
    "Some of the companies have since shied from appearing as the final word on who gets to say what online. Facebook referred Mr. Trump’s case to its Oversight Board, a company-appointed panel of academics, journalists and former members of government. The board ruled that Facebook was right to bar Mr. Trump, but it said the company had not thoroughly explained its decision and should revisit an indefinite suspension.\n",
    "\n",
    "In June, Facebook said Mr. Trump’s ban would last at least two years, keeping the former president off the site through the 2022 midterm elections.\n",
    "\n",
    "A blockbuster deal. Elon Musk, the world’s wealthiest man, capped what seemed an improbable attempt by the famously mercurial billionaire to buy Twitter for roughly $44 billion. Here’s how the deal unfolded:\n",
    "\n",
    "The initial offer. Mr. Musk made an unsolicited bid worth more than $40 billion for the influential social network, saying that he wanted to make Twitter a private company and that he wanted people to be able to speak more freely on the service.\n",
    "\n",
    "The response. Twitter’s board countered Mr. Musk’s offer with a defense mechanism known as a “poison pill.” This well-worn corporate tactic makes a company less palatable to a potential acquirer by making it more expensive for them to buy shares above a certain threshold.\n",
    "\n",
    "Securing financing. Though his original offer had scant details and was received skeptically by Wall Street, Mr. Musk has been moving swiftly to secure commitments worth $46.5 billion to finance his bid, putting pressure on Twitter’s board to take his advances seriously.\n",
    "\n",
    "Striking a deal. With the financing in place, Twitter’s board met with Mr. Musk to discuss his offer. The two sides soon reached a deal, with the social media company agreeing to sell itself for $54.20 a share.\n",
    "\n",
    "What’s next? Shareholders will vote on the offer, which will also be reviewed by regulators. The deal is expected to take three to six months to close. In the meantime, scrutiny is likely to be intense and several questions remain about Mr. Musk’s plans for the company.\n",
    "\n",
    "“Free speech is essential to a functioning democracy. Do you believe Twitter rigorously adheres to this principle?” he asked.\n",
    "\n",
    "At another point, Mr. Musk wondered, “Is a new platform needed?”\n",
    "\n",
    "After Mr. Musk inked the deal to buy Twitter last month, he reiterated his free speech stance and said he would take the company private to improve the service. He added that he hoped to increase trust by making Twitter’s technology more transparent, defeating the bots that spam people on the platform and “authenticating all humans.” He also said he hoped his worst critics would remain on Twitter, because “that is what free speech means.”\n",
    "\n",
    "On Tuesday, he became more specific. “Permanent bans should be extremely rare,” Mr. Musk said, adding that they should be reserved “for accounts that are bots or spam” and “where there’s just no legitimacy to the account at all.”\n",
    "\n",
    "But he also said that “doesn’t mean that somebody gets to say whatever they want to say.” Mr. Musk said he was in favor of temporary suspensions of accounts “if they say something that is illegal or otherwise just, you know, destructive to the world.” He also raised the idea that a particular tweet could be “made invisible or have very limited traction.”\n",
    "\n",
    "Apart from Mr. Trump, others who have been indefinitely barred from Twitter for violating its policies include Representative Marjorie Taylor Greene, Republican of Georgia, the far-right figure Milo Yiannopoulos, and celebrities like Tila Tequila. Twitter also labels tweets that are factually inaccurate or that might incite violence.\n",
    "\n",
    "Inside Twitter on Tuesday, some employees worried that Mr. Musk’s changes would unwind years of work on the company’s policies and unravel millions of dollars of investment in content moderation to stem abuse on the platform, four current and former employees said. Some said they hoped Mr. Musk would lose interest in the site, while others have begun reaching out to recruiters and friends at other tech companies for new opportunities.\n",
    "\n",
    "Still others were excited at the prospect of Mr. Musk in charge, the current and former employees said. Mr. Musk has pitched investors on quintupling Twitter’s revenue and on the service topping more than 900 million users by 2028, up from 217 million or so today.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input X tokenized. Generating Summary ...\n",
      "Summary Generated. Decoding Summary ...\n",
      "Summary Decoded.\n",
      "['Elon Musk said on Tuesday that he would “reverse the permanent ban” of former President Donald J. Trump on Twitter and let him back on the social network, in one of the first specific comments by Mr. Musk, who struck a deal last month to buy Twitter for $44 billion, said at a Financial Times conference that the company’s decision to bar Mr. The billionaire, who also leads the electric carmaker Tesla and the rocket company SpaceX, has called himself a “free speech absolutist” and has said he is unhappy with how Twitter decides what can and cannot be posted online.']\n",
      "CPU times: user 33.5 s, sys: 5.31 s, total: 38.8 s\n",
      "Wall time: 29.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "text_summary = get_summary(pegasus_tokenizer, pegasus_large, test_text)\n",
    "print(text_summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input X tokenized. Generating Summary ...\n",
      "Summary Generated. Decoding Summary ...\n",
      "Summary Decoded.\n",
      "['Will reverse permanent ban on Trump on Twitter: Elon Musk']\n",
      "CPU times: user 2.69 s, sys: 467 ms, total: 3.16 s\n",
      "Wall time: 2.81 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "text_headline = get_summary(pegasus_tokenizer, pegasus_finetuned, text_summary)\n",
    "print(text_headline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': 'user',\n",
       " 'id': '62591773a68a1fd2395f75c4',\n",
       " 'name': 'kubershahi',\n",
       " 'fullname': 'Kuber Shahi',\n",
       " 'email': 'shahikuber97@gmail.com',\n",
       " 'emailVerified': True,\n",
       " 'plan': 'NO_PLAN',\n",
       " 'canPay': False,\n",
       " 'isPro': False,\n",
       " 'periodEnd': None,\n",
       " 'avatarUrl': 'https://aeiljuispo.cloudimg.io/v7/https://cdn-uploads.huggingface.co/production/uploads/1654633500663-62591773a68a1fd2395f75c4.png?w=200&h=200&f=face',\n",
       " 'orgs': [],\n",
       " 'auth': {'type': 'access_token',\n",
       "  'accessToken': {'displayName': 'Jupyter Notebook', 'role': 'write'}}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "whoami()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf_token = HfFolder().get_token()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create_repo(\"pegasus-inshorts\", token=hf_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d77e295c0d64616a0ab495bca2ae6bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "spiece.model:   0%|          | 0.00/1.91M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/kubershahi/pegasus-inshorts/commit/c699129fab6a53dce062df44827b3b117e3b2eaf', commit_message='Upload tokenizer', commit_description='', oid='c699129fab6a53dce062df44827b3b117e3b2eaf', pr_url=None, pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pegasus_tokenizer.push_to_hub(\"pegasus-inshorts\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f71c15582cb44846be649afb27109082",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/2.28G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/kubershahi/pegasus-inshorts/commit/77a417061de65e2ac4d178576250ded5c3e07c33', commit_message='Upload PegasusForConditionalGeneration', commit_description='', oid='77a417061de65e2ac4d178576250ded5c3e07c33', pr_url=None, pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pegasus_finetuned.push_to_hub(\"pegasus-inshorts\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "colab": {
   "authorship_tag": "ABX9TyOx+LFhBD37vffFG7idB6n/",
   "collapsed_sections": [],
   "name": "project.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "work",
   "language": "python",
   "name": "work"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}