Week_05_(1)_Scraping_Twitter_data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Week 06 (1) Scraping Twitter data.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFNwrV7QLrlh"
      },
      "source": [
        "# **Scraping data from Twitter using Python package**\n",
        "\n",
        "This tutorial will follow through the process of scraping tweets from Twitter using Twint python library.  \n",
        "\n",
        "Twint is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API. Twint utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends.\n",
        "\n",
        "Twint documentation can be found from [this URL](https://github.com/twintproject/twint)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "StpJjsogVeBf"
      },
      "source": [
        "## Install and Load Libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RjqmeVAeRpOf"
      },
      "source": [
        "**twint** package is not pre-installed in the Colab environment. Therefore run the following command to install it.  \n",
        "If you are working on your own Python environment, you can install twint using Anaconda interactive user interface or running *pip install twint* from your command prompt."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tilDnFIuLmuW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "186644b9-3981-441a-9e8d-b7325852cf33"
      },
      "source": [
        "!pip install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint\n",
        "# You need to go to Runtime option and select restart runtime to get the effect of this installation."
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Obtaining twint from git+https://github.com/twintproject/twint.git@origin/master#egg=twint\n",
            "  Cloning https://github.com/twintproject/twint.git (to revision origin/master) to ./src/twint\n",
            "  Running command git clone -q https://github.com/twintproject/twint.git /content/src/twint\n",
            "\u001b[33m  WARNING: Did not find branch or tag 'origin/master', assuming revision or ref.\u001b[0m\n",
            "  Running command git checkout -q origin/master\n",
            "Collecting aiohttp\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/88/c0/5890b4c8b04a79b7360e8fe4490feb0bb3ab179743f199f0e6220cebd568/aiohttp-3.7.4.post0-cp37-cp37m-manylinux2014_x86_64.whl (1.3MB)\n",
            "\u001b[K     |████████████████████████████████| 1.3MB 5.7MB/s \n",
            "\u001b[?25hCollecting aiodns\n",
            "  Downloading https://files.pythonhosted.org/packages/da/01/8f2d49b441573fd2478833bdba91cf0b853b4c750a1fbb9e98de1b94bb22/aiodns-2.0.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied, skipping upgrade: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from twint) (4.6.3)\n",
            "Collecting cchardet\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/80/72/a4fba7559978de00cf44081c548c5d294bf00ac7dcda2db405d2baa8c67a/cchardet-2.1.7-cp37-cp37m-manylinux2010_x86_64.whl (263kB)\n",
            "\u001b[K     |████████████████████████████████| 266kB 16.6MB/s \n",
            "\u001b[?25hCollecting dataclasses\n",
            "  Downloading https://files.pythonhosted.org/packages/26/2f/1095cdc2868052dd1e64520f7c0d5c8c550ad297e944e641dbf1ffbb9a5d/dataclasses-0.6-py3-none-any.whl\n",
            "Collecting elasticsearch\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/09/93/461a042becf2a35a666fb7dbb2fa31f0f766dfd1b01e7d971f4ad51f0d69/elasticsearch-7.12.0-py2.py3-none-any.whl (334kB)\n",
            "\u001b[K     |████████████████████████████████| 337kB 16.5MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: pysocks in /usr/local/lib/python3.7/dist-packages (from twint) (1.7.1)\n",
            "Requirement already satisfied, skipping upgrade: pandas in /usr/local/lib/python3.7/dist-packages (from twint) (1.1.5)\n",
            "Collecting aiohttp_socks\n",
            "  Downloading https://files.pythonhosted.org/packages/9a/6c/d302e5a8097fee1e83b9f8e9da10d7752fbf27c74db18b3cc9528b3479be/aiohttp_socks-0.6.0-py3-none-any.whl\n",
            "Collecting schedule\n",
            "  Downloading https://files.pythonhosted.org/packages/43/8c/74529fcfbfaa93b8e88ba5d7f883805f87ee167ad878d0638cc012e5acc0/schedule-1.0.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied, skipping upgrade: geopy in /usr/local/lib/python3.7/dist-packages (from twint) (1.17.0)\n",
            "Collecting fake-useragent\n",
            "  Downloading https://files.pythonhosted.org/packages/d1/79/af647635d6968e2deb57a208d309f6069d31cb138066d7e821e575112a80/fake-useragent-0.1.11.tar.gz\n",
            "Collecting googletransx\n",
            "  Downloading https://files.pythonhosted.org/packages/27/e1/77cd530afec7944d40c5bdd260bcc111be4012b045c82d4e3ffec90b2a42/googletransx-2.4.2.tar.gz\n",
            "Collecting multidict<7.0,>=4.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7c/a6/4123b8165acbe773d1a8dc8e3f0d1edea16d29f7de018eda769abb56bd30/multidict-5.1.0-cp37-cp37m-manylinux2014_x86_64.whl (142kB)\n",
            "\u001b[K     |████████████████████████████████| 143kB 13.3MB/s \n",
            "\u001b[?25hCollecting yarl<2.0,>=1.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f1/62/046834c5fc998c88ab2ef722f5d42122230a632212c8afa76418324f53ff/yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294kB)\n",
            "\u001b[K     |████████████████████████████████| 296kB 15.7MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: typing-extensions>=3.6.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (3.7.4.3)\n",
            "Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (20.3.0)\n",
            "Collecting async-timeout<4.0,>=3.0\n",
            "  Downloading https://files.pythonhosted.org/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl\n",
            "Requirement already satisfied, skipping upgrade: chardet<5.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (3.0.4)\n",
            "Collecting pycares>=3.0.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d3/ea/6367930636a9859cc9ea8cb738a3e0c4f9929c596806214d5df4c48565db/pycares-3.1.1-cp37-cp37m-manylinux2010_x86_64.whl (228kB)\n",
            "\u001b[K     |████████████████████████████████| 235kB 19.0MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: urllib3<2,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint) (1.24.3)\n",
            "Requirement already satisfied, skipping upgrade: certifi in /usr/local/lib/python3.7/dist-packages (from elasticsearch->twint) (2020.12.5)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.15.4 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (1.19.5)\n",
            "Requirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (2.8.1)\n",
            "Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (2018.9)\n",
            "Collecting python-socks[asyncio]>=1.2.2\n",
            "  Downloading https://files.pythonhosted.org/packages/b8/4c/8376caa39edfb245b2dc6044f7238cf9bc1fe8225bdbb23ea142a24d1ec8/python_socks-1.2.3-py3-none-any.whl\n",
            "Requirement already satisfied, skipping upgrade: geographiclib<2,>=1.49 in /usr/local/lib/python3.7/dist-packages (from geopy->twint) (1.50)\n",
            "Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.7/dist-packages (from googletransx->twint) (2.23.0)\n",
            "Requirement already satisfied, skipping upgrade: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl<2.0,>=1.0->aiohttp->twint) (2.10)\n",
            "Requirement already satisfied, skipping upgrade: cffi>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pycares>=3.0.0->aiodns->twint) (1.14.5)\n",
            "Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->twint) (1.15.0)\n",
            "Requirement already satisfied, skipping upgrade: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.5.0->pycares>=3.0.0->aiodns->twint) (2.20)\n",
            "Building wheels for collected packages: fake-useragent, googletransx\n",
            "  Building wheel for fake-useragent (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fake-useragent: filename=fake_useragent-0.1.11-cp37-none-any.whl size=13485 sha256=22de0fc6ba80fdfca0a91961ba10eb13184490b99d657c21e474872e33cf7190\n",
            "  Stored in directory: /root/.cache/pip/wheels/5e/63/09/d1dc15179f175357d3f5c00cbffbac37f9e8690d80545143ff\n",
            "  Building wheel for googletransx (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for googletransx: filename=googletransx-2.4.2-cp37-none-any.whl size=15970 sha256=25f9a7b21233f8723a3382bd3ae0e1d3b6a30c97a56c3bd13d070abb8a2faab3\n",
            "  Stored in directory: /root/.cache/pip/wheels/04/63/5f/75e7e94eb62517946116a783e4cd8970c4789c990bbc732616\n",
            "Successfully built fake-useragent googletransx\n",
            "Installing collected packages: multidict, yarl, async-timeout, aiohttp, pycares, aiodns, cchardet, dataclasses, elasticsearch, python-socks, aiohttp-socks, schedule, fake-useragent, googletransx, twint\n",
            "  Running setup.py develop for twint\n",
            "Successfully installed aiodns-2.0.0 aiohttp-3.7.4.post0 aiohttp-socks-0.6.0 async-timeout-3.0.1 cchardet-2.1.7 dataclasses-0.6 elasticsearch-7.12.0 fake-useragent-0.1.11 googletransx-2.4.2 multidict-5.1.0 pycares-3.1.1 python-socks-1.2.3 schedule-1.0.0 twint yarl-1.6.3\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vyjO-aGjRvUn"
      },
      "source": [
        "import twint\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11eYw4aoWV6d"
      },
      "source": [
        "## Design the scaping parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJJf66ZgMbCm"
      },
      "source": [
        "Next you need to design which parameters you need to scrape. The parameters to be customised can be found from Twint API located in [this URL](https://github.com/twintproject/twint/wiki/Configuration).  \n",
        "\n",
        "For this exercise, we will scrape tweets with specific keywords and within a pre-given time period."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1UnCO28zXdce"
      },
      "source": [
        "keyword = 'LaTrobeBusiness'"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G1B4KTxlXdfN"
      },
      "source": [
        "since_date = \"2020-03-01 00:00:00\"  # Should be '%Y-%m-%d %H:%M:%S' time format"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pb8bC4g7XnI8"
      },
      "source": [
        "Set the parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4tX8qTPPXdiI"
      },
      "source": [
        "c = twint.Config()\n",
        "c.Search = keyword\n",
        "c.Since = since_date"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znpZxw73cFPr"
      },
      "source": [
        "Output parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6eg2OLIxcD0T"
      },
      "source": [
        "c.Lang = \"en\"\n",
        "c.Store_csv = True\n",
        "c.Output = keyword"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJcve_ANbZvw"
      },
      "source": [
        "Format the output attributes.  \n",
        "Available parameters can be found from [this URL](https://github.com/twintproject/twint/wiki/Tweet-attributes)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G2AAxKx7XzAu"
      },
      "source": [
        "Here we will use exeption handling in order to warn us if there is an error. This is called python exception handling using try-catch block."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ryE6s7uGXq-o"
      },
      "source": [
        "try:\n",
        "  # Start search\n",
        "  twint.run.Search(c)\n",
        "except Exception as e:\n",
        "  print(e)\n",
        "  print('Error in {}'.format(keyword))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9pgnAEiNVPkT"
      },
      "source": [
        "The data will be saved in different folders based on the keywords in the **Files** tab.  \n",
        "To download, right click on tweets.csv file, and download."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R29fdD45eXgA"
      },
      "source": [
        "### Keyword combinations\n",
        "\n",
        "You can search and scrape tweets using multiple keywords and combinations as documented in [this URL](https://github.com/twintproject/twint/issues/165#issuecomment-399747575)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J-t7kMz8e1OQ"
      },
      "source": [
        "keyword = 'latrobe covid19'\n",
        "since_date = \"2020-03-01 00:00:00\"  # Should be '%Y-%m-%d %H:%M:%S' time format\n",
        "c = twint.Config()\n",
        "c.Search = keyword\n",
        "c.Limit = 40  # (Increments of 20) e.g., If 23, it will scrape 40. If 12, it will scrape 20.\n",
        "c.Since = since_date\n",
        "c.Store_csv = True\n",
        "c.Output = keyword"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qI6_JL-CfEzF"
      },
      "source": [
        "try:\n",
        "  # Start search\n",
        "  twint.run.Search(c)\n",
        "except Exception as e:\n",
        "  print(e)\n",
        "  print('Error in {}'.format(keyword))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Noa8UV3hghu0"
      },
      "source": [
        "### Multiple keywords\n",
        "\n",
        "We can loop to scape multiple keywords."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gulilp79fE5d"
      },
      "source": [
        "Keyword_list = ['LBS', 'data analytics latrobe']"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WUp9HUflXAFh"
      },
      "source": [
        "since_date = \"2020-03-01 00:00:00\"  # Should be '%Y-%m-%d %H:%M:%S' time format"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZjBvd-gNEja"
      },
      "source": [
        "for key in Keyword_list:\n",
        "\n",
        "  print('Searching Tweets for {}'.format(key))\n",
        "\n",
        "  c = twint.Config()\n",
        "  c.Search = key\n",
        "  c.Limit = 40\n",
        "  c.Since = since_date\n",
        "  c.Store_csv = True\n",
        "  c.Output = key\n",
        "\n",
        "  try:\n",
        "    # Start search\n",
        "    twint.run.Search(c)\n",
        "  except Exception as e:\n",
        "    print(e)\n",
        "    print('Error in {}'.format(key))\n",
        "  continue\n",
        "\n",
        "  print('\\n\\nCompleted {}\\n\\n'.format(key))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}