Week 1 Model Implementation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coding Method Explanations for Qiang"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation\n",
    "\n",
    "I use the code dataset.py from the Stanford Fake News codes, as I have not personally identified any differences in our pre-processing requirements to theirs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the Graph\n",
    "\n",
    "Again, I almost exactly use the Stanford code generate_graphs.py. I have made the graph dynamic by creating an array that appends the collection of edges each time a new edge is formed. This means we have an array 'edges', which stores the current graph snapshot and an array 'dynamic_graph', which stores each timestamped version of 'edges'. \n",
    "\n",
    "I show the two adjusted functions below. My version of the Stanford code generate_graphs.py is simply titled graphs.py and is among the other documents I have sent you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_dynamic_graph(dumps_dir, dataset, timestamp):\n",
    "    \"\"\"\n",
    "    :return: simple event graph, directed, unweighted, with edges created up to current timestamp\n",
    "             dynamic graph, as a sequence of graph snapshots taken each time a new event node is added\n",
    "    \"\"\"\n",
    "    nodes = {}\n",
    "    edges = []\n",
    "    dynamic_graph = []\n",
    "    num_nodes = 0\n",
    "    dataset_dir = os.path.join(dumps_dir, dataset)\n",
    "    for data_point in twitter_tree_iterator():\n",
    "        if data_point['time_out'] < timestamp:\n",
    "            node_in, node_out = data_point['node_in'], data_point['node_out']\n",
    "            if node_in not in nodes:\n",
    "                nodes[node_in] = num_nodes\n",
    "                num_nodes += 1\n",
    "            if node_out not in nodes:\n",
    "                nodes[node_out] = num_nodes\n",
    "                num_nodes += 1\n",
    "            edges.append([nodes[node_in], nodes[node_out]])\n",
    "            dynamic_graph.append(edges)\n",
    "\n",
    "    return edges, dynamic_graph\n",
    "\n",
    "def generate_datum(dynamic_graph):\n",
    "    \"\"\"\n",
    "    :return: datum of form (claim, c; list of engagements, S; dynamic graph, G)\n",
    "    \"\"\"\n",
    "    claim = dynamic_graph[0][0][0]\n",
    "    engagements = dynamic_graph[-1][1:]\n",
    "    datum = [claim, engagements, dynamic_graph]\n",
    "    \n",
    "    return datum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder: Temporal Graph Sum \n",
    "\n",
    "In the document temporal_graph_sum.py, I attempt to implement equations (1) and (2) of the overleaf document. I think that equation (1) is more likely to be correctly implemented, but I am unsure how to include $\\phi$, the time embedding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eq_1():\n",
    "    h = []\n",
    "    for i in dynamic_graph:\n",
    "        h.append(np.zeros_like(dynamic_graph[i]))\n",
    "    h_tilde = np.zeros_like(h)\n",
    "    del h_tilde[0]\n",
    "    h[0] = dynamic_graph[0]\n",
    "    for i in h_tilde:\n",
    "        for j in h_tilde[i]:\n",
    "            h_tilde[i][j] = np.sum(np.concatenate(h[i][j],edges[j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I then attempt to implement the MLP of equation 2, such that the encoder roughly looks as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "class TGS_stack(torch.nn.Module):\n",
    "    \n",
    "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
    "        super(TGS_stack, self).__init__()\n",
    "        self.input_dim = input_dim\n",
    "        self.hidden_dim  = hidden_dim\n",
    "        self.fc1 = torch.nn.Linear(self.input_size, self.hidden_size)\n",
    "        self.relu = torch.nn.ReLU()\n",
    "        self.fc2 = torch.nn.Linear(self.hidden_size, 1)\n",
    "        self.sigmoid = torch.nn.Sigmoid()\n",
    "    \n",
    "    \n",
    "    def node_embeddings(dynamic_graph):\n",
    "        h = []\n",
    "        for i in dynamic_graph:\n",
    "            h.append(np.zeros_like(dynamic_graph[i]))\n",
    "        h_tilde = np.zeros_like(h)\n",
    "        del h_tilde[0]\n",
    "        h[0] = dynamic_graph[0]\n",
    "        for i in h_tilde:\n",
    "            for j in h_tilde[i]:\n",
    "                h_tilde[i][j] = np.sum(np.concatenate(h[i][j],edges[j]))\n",
    "            \n",
    "        def forward(self, x):\n",
    "            hidden = self.fc1(x)\n",
    "            relu = self.relu(hidden)\n",
    "            output = self.fc2(relu)\n",
    "            output = self.sigmoid(output)\n",
    "            return output\n",
    "            \n",
    "        for i in h:\n",
    "            for j in h[i]:\n",
    "                h[i+1][j] = forward(np.concatenate(h[i][j],h_tilde[i][j]))\n",
    "                \n",
    "        return h\n",
    "    \n",
    "    def generate_hidden(h):\n",
    "        hidden = []\n",
    "        for i in h[-1]:\n",
    "            hidden.append(h[-1][i])\n",
    "            \n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This currently only uses a single-layer perceptron. I am unsure how to extend it to multi-layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Claim Veracity Prediction\n",
    "\n",
    "Here, I attempt to implement equations (14) and (15) of the overleaf document. I have spent lots of time reading the literature on graph learning algorithms as well as pytorch documentation, so I understand the concepts quite well, but given my lack of experience, I am still struggling with transferring that into the correct code. My very rough idea of how to execute this task is shown below and in veracity_prediction.py file I have sent you. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class veracity(torch.nn.Module):\n",
    "    \n",
    "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
    "        super(veracity, self).__init__()\n",
    "        self.input_dim = input_dim\n",
    "        self.hidden_dim  = hidden_dim\n",
    "        self.relu = torch.nn.ReLU()\n",
    "        self.weight = torch.nn.Parameter(data=torch.randn(len(dynamic_graph[-1])), requires_grad=True)\n",
    "        self.bias = torch.nn.Parameter(data=torch.Tensor(0,0), requires_grad=False)\n",
    "        \n",
    "    def forward(self,x):\n",
    "        hidden = x*self.weight + self.bias\n",
    "        relu = self.relu(hidden)\n",
    "        soft = np.exp(relu - np.max(relu))\n",
    "        output = soft/soft.sum()\n",
    "        return output\n",
    "    \n",
    "    def compute_loss(self,output,veracity_label):\n",
    "        y_1 = output[0]\n",
    "        y_2 = output[1]\n",
    "        loss = -veracity_label*np.log(y_2) - (1-veracity_label)*np.log(1 - y_1)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I am particularly unsure how to initialise the learnable weight matrix and bias term. Perhaps it would be beneficial for you to point me along some more explicit lines if you do not think my implementations very appropriate. Or if you think I am along the right lines, I am happy to continue with these approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further Work\n",
    "\n",
    "I will use your codes for the Hawkes process method and implement the timestamp prediction and stance classification tasks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}