Hydrate_TweetIDs_Arabic_Fake_News_tweets.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Hydrate_TweetIDs_Arabic_Fake_News_tweets.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XTcCAKlavovi"
      },
      "source": [
        "\n",
        "\n",
        "This notebook helps in retriving the full object of the tweet using TweetID from Arabic-Fake-News-Tweets-Dataset repository on Github using TWARC library.\n",
        "\n",
        "Note: you must have Tiwtter developer account.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-khXVvDStnJQ"
      },
      "source": [
        "**Install** **twarc** **library**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pRsFUXeV-vvs"
      },
      "source": [
        "%pip install twarc\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nTACtP4tuyTO"
      },
      "source": [
        "**Mount** **Drive**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BW2_Znl__R5y"
      },
      "source": [
        "from google.colab import drive \n",
        "from twarc import Twarc\n",
        "import pandas as pd\n",
        "import  json\n",
        "import csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UAdPYfXa_R-E"
      },
      "source": [
        "drive.mount(\"/content/drive\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCur1FCmu3oe"
      },
      "source": [
        "**Twitter API Keys**\n",
        "\n",
        "Once you've created your application, write down the consumer key, consumer secret, access token and access token secret\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "apGNv7XsviGl"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVsVK3NY_SGS"
      },
      "source": [
        "#Variables that contains the user credentials to access Twitter API \n",
        "access_token = \"\" #@param {type:\"string\"}\n",
        "access_token_secret =\"\" #@param {type:\"string\"}\n",
        "consumer_key = \"\" #@param {type:\"string\"}\n",
        "consumer_secret =\"\" #@param {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7tpAYJ3x_SOM"
      },
      "source": [
        "T = Twarc(consumer_key, consumer_secret, access_token, access_token_secret)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NLx5IHWOu_h1"
      },
      "source": [
        "**`Hydrate TweetID`**\n",
        "\n",
        "Set up your output and input files\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TOEflcks_SUV"
      },
      "source": [
        "Inputfile= \"\" #@param {type:\"string\"}\n",
        "OutputFile= \"\" #@param {type:\"string\"}\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Xs8CDv0vWFv"
      },
      "source": [
        "The following code will hydrate the ides and retrieve json of the tweet. note that the time of this code depends on the number of the tweet you want to hydrate. Due to the Twitter API limit, the code will go to sleep for a few minutes once the limit is reached and automatically continue."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P1FvS8Wa_SaX"
      },
      "source": [
        "ids = []\n",
        "HydratedTweets=[]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SvJYwI0s_ShE"
      },
      "source": [
        "with open(Inputfile, \"r\") as ids_file: #open the input file\n",
        "     ids=ids_file.read().split() #read and store the ids  \n",
        "count = len(HydratedTweets) #set up counter\n",
        "end_index =len(ids) \n",
        "print(\"Total IDs: \" + str(len(ids)))\n",
        "for tweet in T.hydrate(iter(ids)): \n",
        "    HydratedTweets.append(tweet)  #appent tweet json into a list \n",
        "    count += 1      \n",
        "    if (count == end_index): #stop condition \n",
        "        break\n",
        "print(str(len(HydratedTweets))+ \"Tweet Saved\")\n",
        "\n",
        "with open('JsonOutputFile.txt', 'w') as outfile: #open json file\n",
        "    json.dump(HydratedTweets, outfile) # write the data into json file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzxDwOEOva4F"
      },
      "source": [
        "This code Store the data into CSV file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RGScEH8__SqP"
      },
      "source": [
        "fileInput = \"JsonOutputFile.txt\"\n",
        "inputFile = open(fileInput) #open json file\n",
        "outputFile = open(OutputFile, 'w') #load csv file\n",
        "data = json.load(inputFile) #load json content\n",
        "inputFile.close() #close the input file\n",
        "output = csv.writer(outputFile) #create a csv.write\n",
        "output.writerow(data[0].keys()) # header row\n",
        "for row in data:\n",
        "    output.writerow(row.values()) #values row"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d9t4ZlDz_SwR"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}