ejercicio_8_v2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ejercicio_8_v2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1IQtU0JNySR8_7nO9fTDOj70bQR9whU1q",
      "authorship_tag": "ABX9TyN1GczddBKVmJwMgIGIPIIA",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alejovillores/tp2_spark_villores_alejo/blob/main/ejercicio_8_v2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zffuRmBkz_Sj"
      },
      "source": [
        "## Ejercicio 8\n",
        "Considerando el pagelink_sample.csv, usando una representación de grafos obtener\n",
        "aquellos contenidos que tienen “relaciones no correspondidas”. Entendemos como\n",
        "funciona una relación correspondida con un ejemplo: Si el contenido A tiene un link\n",
        "al B, pero B no tiene un link a A, podemos decir que B tiene una relación no\n",
        "correspondida con A."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tCCLLGIEzzWD",
        "outputId": "d9699436-7fb5-4b1f-c4ff-0f00c5c984bd"
      },
      "source": [
        "!pip install pyspark\n",
        "!pip install -U -q PyDrive\n",
        "!apt update\n",
        "!apt install openjdk-8-jdk-headless -qq\n",
        "#!apt install default-jre\n",
        "#!apt install default-jdk\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pyspark\n",
            "  Downloading pyspark-3.2.0.tar.gz (281.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 281.3 MB 37 kB/s \n",
            "\u001b[?25hCollecting py4j==0.10.9.2\n",
            "  Downloading py4j-0.10.9.2-py2.py3-none-any.whl (198 kB)\n",
            "\u001b[K     |████████████████████████████████| 198 kB 60.0 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.2.0-py2.py3-none-any.whl size=281805912 sha256=ec1436334ef24c06cb76fc95796fb76950fe2cf42ec27317935f808894dd3b84\n",
            "  Stored in directory: /root/.cache/pip/wheels/0b/de/d2/9be5d59d7331c6c2a7c1b6d1a4f463ce107332b1ecd4e80718\n",
            "Successfully built pyspark\n",
            "Installing collected packages: py4j, pyspark\n",
            "Successfully installed py4j-0.10.9.2 pyspark-3.2.0\n",
            "Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "Ign:2 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [696 B]\n",
            "Hit:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "Get:5 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]\n",
            "Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]\n",
            "Get:7 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n",
            "Get:8 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]\n",
            "Hit:10 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "Get:11 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n",
            "Hit:12 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n",
            "Get:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [786 kB]\n",
            "Hit:14 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease\n",
            "Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n",
            "Get:16 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]\n",
            "Get:17 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,398 kB]\n",
            "Get:18 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,810 kB]\n",
            "Get:19 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,434 kB]\n",
            "Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,213 kB]\n",
            "Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,835 kB]\n",
            "Get:22 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [927 kB]\n",
            "Get:23 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [44.7 kB]\n",
            "Fetched 12.7 MB in 4s (3,015 kB/s)\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "61 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'apt autoremove' to remove it.\n",
            "The following additional packages will be installed:\n",
            "  openjdk-8-jre-headless\n",
            "Suggested packages:\n",
            "  openjdk-8-demo openjdk-8-source libnss-mdns fonts-dejavu-extra\n",
            "  fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei\n",
            "  fonts-wqy-zenhei fonts-indic\n",
            "The following NEW packages will be installed:\n",
            "  openjdk-8-jdk-headless openjdk-8-jre-headless\n",
            "0 upgraded, 2 newly installed, 0 to remove and 61 not upgraded.\n",
            "Need to get 36.5 MB of archives.\n",
            "After this operation, 143 MB of additional disk space will be used.\n",
            "Selecting previously unselected package openjdk-8-jre-headless:amd64.\n",
            "(Reading database ... 155062 files and directories currently installed.)\n",
            "Preparing to unpack .../openjdk-8-jre-headless_8u292-b10-0ubuntu1~18.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jre-headless:amd64 (8u292-b10-0ubuntu1~18.04) ...\n",
            "Selecting previously unselected package openjdk-8-jdk-headless:amd64.\n",
            "Preparing to unpack .../openjdk-8-jdk-headless_8u292-b10-0ubuntu1~18.04_amd64.deb ...\n",
            "Unpacking openjdk-8-jdk-headless:amd64 (8u292-b10-0ubuntu1~18.04) ...\n",
            "Setting up openjdk-8-jre-headless:amd64 (8u292-b10-0ubuntu1~18.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode\n",
            "Setting up openjdk-8-jdk-headless:amd64 (8u292-b10-0ubuntu1~18.04) ...\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rDNVqaWj0Cr3"
      },
      "source": [
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "from pyspark.sql import *\n",
        "from pyspark.sql.functions import *\n",
        "from pyspark import SparkContext\n",
        "from pyspark.sql import SQLContext\n",
        "import pandas as pd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V5hdu2Nn0Xdt",
        "outputId": "9bcf44b7-4a7f-4957-c8b8-f272308c5279"
      },
      "source": [
        "!pip install pyarrow"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.7/dist-packages (3.0.0)\n",
            "Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from pyarrow) (1.19.5)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "82rysU3m0cJX"
      },
      "source": [
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ytizDTzw0gVj"
      },
      "source": [
        "spark = SparkSession.builder.getOrCreate()\n",
        "sc = spark.sparkContext"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "57Moly6H0qwT",
        "outputId": "ac28f452-57c5-4bb8-8fa6-6c1a06eacae0"
      },
      "source": [
        "pagelinks = pd.read_parquet('/content/drive/MyDrive/2C2021/Datos/Trabajo Practico 2/pagelinks_sample.csv')\n",
        "pagelinks.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>pl_from</th>\n",
              "      <th>pl_namespace</th>\n",
              "      <th>pl_title</th>\n",
              "      <th>pl_from_namespace</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4840492</td>\n",
              "      <td>10</td>\n",
              "      <td>Bandera_cÃ³digo_del_COI_TPE</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>950948</td>\n",
              "      <td>0</td>\n",
              "      <td>Esteban_DuÅ¡an</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6418187</td>\n",
              "      <td>0</td>\n",
              "      <td>Insecta</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3436623</td>\n",
              "      <td>0</td>\n",
              "      <td>Pleasure_to_Kill</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4713066</td>\n",
              "      <td>0</td>\n",
              "      <td>DamiÃ¡n_Oliver</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   pl_from  pl_namespace                     pl_title  pl_from_namespace\n",
              "0  4840492            10  Bandera_cÃ³digo_del_COI_TPE                  2\n",
              "1   950948             0               Esteban_DuÅ¡an                  0\n",
              "2  6418187             0                      Insecta                  0\n",
              "3  3436623             0             Pleasure_to_Kill                  0\n",
              "4  4713066             0               DamiÃ¡n_Oliver                  0"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 686
        },
        "id": "4cK6Gfkg028i",
        "outputId": "d19a3b61-883a-4c62-d6cd-8290688eea95"
      },
      "source": [
        "contents = pd.read_parquet('/content/drive/MyDrive/2C2021/Datos/Trabajo Practico 2/contents.csv')\n",
        "contents"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>id</th>\n",
              "      <th>namespace</th>\n",
              "      <th>revision_id</th>\n",
              "      <th>parent_revision</th>\n",
              "      <th>revision_timestamp</th>\n",
              "      <th>revisor_username</th>\n",
              "      <th>revisor_id</th>\n",
              "      <th>revisor_ip</th>\n",
              "      <th>revisor_comment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wikipedia:Artículos solicitados</td>\n",
              "      <td>5</td>\n",
              "      <td>4</td>\n",
              "      <td>132533307</td>\n",
              "      <td>132516631.0</td>\n",
              "      <td>2021-01-19T18:40:34Z</td>\n",
              "      <td>Metrónomo</td>\n",
              "      <td>1708233.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Así esta mejor</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Andorra</td>\n",
              "      <td>7</td>\n",
              "      <td>0</td>\n",
              "      <td>138035057</td>\n",
              "      <td>138027475.0</td>\n",
              "      <td>2021-08-31T18:27:01Z</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2800:CD0:FC1B:A900:C5A4:9E81:7175:2ED2</td>\n",
              "      <td>Añadir contenido</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Argentina</td>\n",
              "      <td>10</td>\n",
              "      <td>0</td>\n",
              "      <td>137966826</td>\n",
              "      <td>137825165.0</td>\n",
              "      <td>2021-08-28T17:17:54Z</td>\n",
              "      <td>Traitrich</td>\n",
              "      <td>4642283.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Reemplazos con [[Usuario:Benjavalero/Replacer|...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Geografía de Andorra</td>\n",
              "      <td>15</td>\n",
              "      <td>0</td>\n",
              "      <td>137986788</td>\n",
              "      <td>133666383.0</td>\n",
              "      <td>2021-08-29T17:16:00Z</td>\n",
              "      <td>Lojwe</td>\n",
              "      <td>5338508.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Reemplazos con [[Usuario:Benjavalero/Replacer|...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Demografía de Andorra</td>\n",
              "      <td>17</td>\n",
              "      <td>0</td>\n",
              "      <td>130869554</td>\n",
              "      <td>129413188.0</td>\n",
              "      <td>2020-11-12T19:56:15Z</td>\n",
              "      <td>Jialxv</td>\n",
              "      <td>3615366.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Correcciones ortográficas con [[Usuario:Benjav...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4132159</th>\n",
              "      <td>Cruz nestoriana</td>\n",
              "      <td>10010593</td>\n",
              "      <td>0</td>\n",
              "      <td>138052016</td>\n",
              "      <td>138051910.0</td>\n",
              "      <td>2021-09-01T12:06:00Z</td>\n",
              "      <td>Copydays</td>\n",
              "      <td>474907.0</td>\n",
              "      <td>None</td>\n",
              "      <td>termino traducción y quito plantilla en desarr...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4132160</th>\n",
              "      <td>Manifestación del orgullo gay de 1977 en Barce...</td>\n",
              "      <td>10010595</td>\n",
              "      <td>0</td>\n",
              "      <td>138051859</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2021-09-01T11:52:28Z</td>\n",
              "      <td>Crls9011</td>\n",
              "      <td>4032712.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Crls9011 trasladó la página [[Manifestación de...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4132161</th>\n",
              "      <td>Rocasaurus</td>\n",
              "      <td>10010596</td>\n",
              "      <td>0</td>\n",
              "      <td>138051902</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2021-09-01T11:57:06Z</td>\n",
              "      <td>Gabriel Solo</td>\n",
              "      <td>158577.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Gabriel Solo trasladó la página [[Rocasaurus]]...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4132162</th>\n",
              "      <td>Medalla Conmemorativa del 50.º Aniversario de ...</td>\n",
              "      <td>10010598</td>\n",
              "      <td>0</td>\n",
              "      <td>138051961</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2021-09-01T12:01:38Z</td>\n",
              "      <td>Términus</td>\n",
              "      <td>5429951.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Términus trasladó la página [[Medalla Conmemor...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4132163</th>\n",
              "      <td>Fatikroy</td>\n",
              "      <td>10010600</td>\n",
              "      <td>0</td>\n",
              "      <td>138051985</td>\n",
              "      <td>138051982.0</td>\n",
              "      <td>2021-09-01T12:03:14Z</td>\n",
              "      <td>Carlukas</td>\n",
              "      <td>86975.0</td>\n",
              "      <td>None</td>\n",
              "      <td>Añadiendo la [[Categoría:Localidades de Tripur...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>4132164 rows × 10 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                     title  ...                                    revisor_comment\n",
              "0                          Wikipedia:Artículos solicitados  ...                                     Así esta mejor\n",
              "1                                                  Andorra  ...                                   Añadir contenido\n",
              "2                                                Argentina  ...  Reemplazos con [[Usuario:Benjavalero/Replacer|...\n",
              "3                                     Geografía de Andorra  ...  Reemplazos con [[Usuario:Benjavalero/Replacer|...\n",
              "4                                    Demografía de Andorra  ...  Correcciones ortográficas con [[Usuario:Benjav...\n",
              "...                                                    ...  ...                                                ...\n",
              "4132159                                    Cruz nestoriana  ...  termino traducción y quito plantilla en desarr...\n",
              "4132160  Manifestación del orgullo gay de 1977 en Barce...  ...  Crls9011 trasladó la página [[Manifestación de...\n",
              "4132161                                         Rocasaurus  ...  Gabriel Solo trasladó la página [[Rocasaurus]]...\n",
              "4132162  Medalla Conmemorativa del 50.º Aniversario de ...  ...  Términus trasladó la página [[Medalla Conmemor...\n",
              "4132163                                           Fatikroy  ...  Añadiendo la [[Categoría:Localidades de Tripur...\n",
              "\n",
              "[4132164 rows x 10 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "eZdJTWTf1LPr",
        "outputId": "9ca2eda5-cde3-4ccf-e9e1-1cf6660d9e0b"
      },
      "source": [
        "pagelinks_reduce = pagelinks.loc[pagelinks['pl_title'].isin(contents['title'])]\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>pl_from</th>\n",
              "      <th>pl_namespace</th>\n",
              "      <th>pl_title</th>\n",
              "      <th>pl_from_namespace</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4840492</td>\n",
              "      <td>10</td>\n",
              "      <td>Bandera_cÃ³digo_del_COI_TPE</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>950948</td>\n",
              "      <td>0</td>\n",
              "      <td>Esteban_DuÅ¡an</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6418187</td>\n",
              "      <td>0</td>\n",
              "      <td>Insecta</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3436623</td>\n",
              "      <td>0</td>\n",
              "      <td>Pleasure_to_Kill</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4713066</td>\n",
              "      <td>0</td>\n",
              "      <td>DamiÃ¡n_Oliver</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   pl_from  pl_namespace                     pl_title  pl_from_namespace\n",
              "0  4840492            10  Bandera_cÃ³digo_del_COI_TPE                  2\n",
              "1   950948             0               Esteban_DuÅ¡an                  0\n",
              "2  6418187             0                      Insecta                  0\n",
              "3  3436623             0             Pleasure_to_Kill                  0\n",
              "4  4713066             0               DamiÃ¡n_Oliver                  0"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "71VvvMIj3ruO",
        "outputId": "5f408897-ede1-45ae-e832-0ba3d269ca33"
      },
      "source": [
        "pagelinks.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 80000000 entries, 0 to 79999999\n",
            "Data columns (total 4 columns):\n",
            " #   Column             Dtype \n",
            "---  ------             ----- \n",
            " 0   pl_from            int64 \n",
            " 1   pl_namespace       int64 \n",
            " 2   pl_title           object\n",
            " 3   pl_from_namespace  int64 \n",
            "dtypes: int64(3), object(1)\n",
            "memory usage: 2.4+ GB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "4gg7wrb43G_B",
        "outputId": "6691ccfb-a3e2-4ee2-e686-e56f6bca403a"
      },
      "source": [
        "pagelinks_reduce.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>pl_from</th>\n",
              "      <th>pl_namespace</th>\n",
              "      <th>pl_title</th>\n",
              "      <th>pl_from_namespace</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6418187</td>\n",
              "      <td>0</td>\n",
              "      <td>Insecta</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>6571966</td>\n",
              "      <td>0</td>\n",
              "      <td>IMDb</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>5461523</td>\n",
              "      <td>4</td>\n",
              "      <td>Vandalismo</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>1098340</td>\n",
              "      <td>0</td>\n",
              "      <td>Wikidata</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>6224910</td>\n",
              "      <td>10</td>\n",
              "      <td>Referencias</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    pl_from  pl_namespace     pl_title  pl_from_namespace\n",
              "2   6418187             0      Insecta                  0\n",
              "8   6571966             0         IMDb                  0\n",
              "12  5461523             4   Vandalismo                  3\n",
              "13  1098340             0     Wikidata                  0\n",
              "16  6224910            10  Referencias                  3"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8KbO4Hxy3eWv",
        "outputId": "fa1d8abf-4ba1-431d-d24e-89235a7c05ce"
      },
      "source": [
        "pagelinks_reduce.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 20572932 entries, 2 to 79999997\n",
            "Data columns (total 4 columns):\n",
            " #   Column             Dtype \n",
            "---  ------             ----- \n",
            " 0   pl_from            int64 \n",
            " 1   pl_namespace       int64 \n",
            " 2   pl_title           object\n",
            " 3   pl_from_namespace  int64 \n",
            "dtypes: int64(3), object(1)\n",
            "memory usage: 784.8+ MB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0tyknhZy3yH5"
      },
      "source": [
        "pagelinks_reduce.to_parquet('/content/drive/MyDrive/2C2021/Datos/Trabajo Practico 2/pagelinks_sample.parquet')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y4AVP0897BRZ",
        "outputId": "1490386b-07ef-4c82-9410-6ed2bfab14a7"
      },
      "source": [
        "sqlContext = SQLContext(sc)\n",
        "df_contenidos = sqlContext.read.parquet('/content/drive/MyDrive/2C2021/Datos/Trabajo Practico 2/contents.csv', header=True, inferSchema=True)\n",
        "rdd_contenidos = df_contenidos.rdd"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/pyspark/sql/context.py:79: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
            "  FutureWarning\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uNvPk7Fs9coW",
        "outputId": "f212e9d3-a711-453b-bd8c-3b28b2250c7c"
      },
      "source": [
        "sqlContext = SQLContext(sc)\n",
        "df_pagelink = sqlContext.read.parquet('/content/drive/MyDrive/2C2021/Datos/Trabajo Practico 2/pagelinks_sample.parquet', header=True, inferSchema=True)\n",
        "rdd_pagelinks = df_pagelink.rdd"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/pyspark/sql/context.py:79: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
            "  FutureWarning\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6iJ_nOnQ-oXC"
      },
      "source": [
        "Combierto mis valores de page links a clave valor, donde la clave es el titulo que comparte con contents"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_zE93Cm-9oUF"
      },
      "source": [
        "rdd_pagelink_title = rdd_pagelinks.filter(lambda x: x.pl_from != None).map(lambda x: (x.pl_title,x.pl_from))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ET_1XRBQ_hOv",
        "outputId": "e711e29b-2b28-4cbc-e94f-62bed7ac47b6"
      },
      "source": [
        "rdd_pagelink_title.take(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Insecta', 6418187), ('IMDb', 6571966), ('Vandalismo', 5461523)]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5rK4cAXH-xvy"
      },
      "source": [
        "rdd_contenidos_title_id = rdd_contenidos.filter(lambda x: x.id != None).map(lambda x: (x.title,x.id))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H7-Z6euT_eT8",
        "outputId": "32cb5835-4b04-458f-b09e-f450ee26a373"
      },
      "source": [
        "rdd_contenidos_title_id.take(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Wikipedia:Artículos solicitados', 5), ('Andorra', 7), ('Argentina', 10)]"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZKAhqu1U_oh-"
      },
      "source": [
        "join_data = rdd_pagelink_title.join(rdd_contenidos_title_id)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_gJPBrEIANgG",
        "outputId": "f4722364-61ef-4a69-cfbb-bfb9b4fabebc"
      },
      "source": [
        "join_data.take(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Trochamminoidea', (7313489, 5782285)),\n",
              " ('Trochamminoidea', (6364333, 5782285)),\n",
              " ('Trochamminoidea', (7313440, 5782285))]"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YUnJYvXyAARN"
      },
      "source": [
        "A traves de los nodos del grafo busco lo pedido\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ckIH037a_7Sf"
      },
      "source": [
        "nodos = join_data.map(lambda x: (x[1][0],x[1][1]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dy3p6Dg0DH71",
        "outputId": "dfc0c0c4-0ef3-4bcd-aab0-a35285e478c5"
      },
      "source": [
        "nodos.take(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(596996, 4050588), (9474473, 4050588), (7061488, 4050588)]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sbaKpwu92L-2"
      },
      "source": [
        "nodos_original_ordenado = nodos.map(lambda x: (tuple(sorted(x)),x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VCjgOIPo2bbX",
        "outputId": "cb6f147b-4106-421c-bd5b-e36bf0b32063"
      },
      "source": [
        "nodos_original_ordenado.take(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[((596996, 4050588), (596996, 4050588)),\n",
              " ((4050588, 9474473), (9474473, 4050588)),\n",
              " ((4050588, 7061488), (7061488, 4050588))]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E7KCnB-IDJxP"
      },
      "source": [
        "rdd_relaciones = nodos.map(lambda x: tuple(sorted(x))).map(lambda x:(x,1)).reduceByKey(lambda a,b: a+b).filter(lambda x: x[1] == 1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xS8WVGsnI2-P"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RzmqQje0D6qz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b57d6598-e99d-4d29-a136-77e31b36304d"
      },
      "source": [
        "nodos_original_ordenado.join(rdd_relaciones).take(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[((10256, 1957010), ((1957010, 10256), 1)),\n",
              " ((2587481, 2657647), ((2587481, 2657647), 1)),\n",
              " ((5206, 7089920), ((7089920, 5206), 1)),\n",
              " ((312036, 312866), ((312036, 312866), 1)),\n",
              " ((41397, 7146771), ((7146771, 41397), 1))]"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    }
  ]
}