Notebook 5, Final Scoring Function.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Scoring Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This final notebook demonstrates how all the functions are incoporated together.\n",
    "\n",
    "Firstly, we import the functions that we created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.shingle import *\n",
    "from src.graph import *\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file `port.txt` is a list of Portuguese lexicon. We create a dictionary in the form of a lexeme mapped to its shingle set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "portuguese = open('data/portuguese/port.txt', \"r+\")\n",
    "wordlist = {}\n",
    "for line in portuguese:\n",
    "    word = line.rstrip()\n",
    "    wordlist[word] = two_ends(word, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file `portuguese_train_mrr.txt` contains info about Portuguese - Romanian cognates.\n",
    "\n",
    "We form the data structure something like this then:\n",
    "\n",
    "`data = [(query, shingle-set of query, correct answer)]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = open('data/portuguese/portuguese_train_mrr.txt', \"r+\")\n",
    "pairs = []\n",
    "for line in train:\n",
    "    word = line.rstrip().split('____')\n",
    "    pairs.append((word[0], word[1]))\n",
    "data = [(p[0], two_ends(p[0],2), p[1]) for p in pairs]\n",
    "avgdl = np.mean([len(s[1]) for s in data])\n",
    "print(\"Average is\", avgdl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to 4th notebook initialize the dictionary of lexicons in the `init_dict()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "port_ro_cog = dict()\n",
    "port_ro_noncog = dict()\n",
    "\n",
    "def init_dict():\n",
    "    cogs = open(\"data/portuguese/graph_cognates_freq.txt\", \"r+\")\n",
    "    for line in cogs:\n",
    "        splits = line.rstrip().split()\n",
    "        port_ro_cog[(splits[1], splits[2])] = int(splits[0])\n",
    "    cogs.close()\n",
    "    noncogs = open(\"data/portuguese/graph_noncognates_freq.txt\", \"r+\")\n",
    "    for line in noncogs:\n",
    "        splits = line.rstrip().split()\n",
    "        port_ro_noncog[(splits[1], splits[2])] = int(splits[0])\n",
    "    noncogs.close()\n",
    "\n",
    "init_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the `pi` function that we created in the 4th notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pi(source, target, k = 1.5):\n",
    "    query = two_ends(source, 2) #Your query\n",
    "    document = two_ends(target, 2) #Your document\n",
    "    qd = common_elements(query, document) # q cap d\n",
    "    first = uncommon_elements(query, qd) # q - (q cap d)\n",
    "    second = uncommon_elements(document, qd) # d - (q cap d)\n",
    "    graph = graph_model(first,second)\n",
    "    res = 0 # sum the frequencies in the dictionary\n",
    "    for i in graph:\n",
    "        if i in port_ro_cog:\n",
    "            res += port_ro_cog[i] ** k\n",
    "    return res / len(graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pi(\"aspirat\", \"aspirar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the third notebook, we take the dirichlet functions for the similarity part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shingles = 470751\n",
    "\n",
    "def smooth(intersection, document, mu):\n",
    "    smooth = []\n",
    "    for word in intersection:\n",
    "        prob = 1.0 + np.divide(document.count(word), mu * frequencies[word] / shingles)\n",
    "        smooth.append(np.log10(prob))\n",
    "    smooth = np.array(smooth)\n",
    "    return smooth\n",
    "\n",
    "def dirichlet(query, document, mu = 100.0):\n",
    "    intersection = [word for word in document if word in query] # intersection\n",
    "    add = len(query) * np.log10(np.divide(mu, mu + len(document)))\n",
    "    score = np.dot(tf(intersection, query), smooth(intersection , document, mu)) + add\n",
    "    return score\n",
    "\n",
    "def tf(intersection, document):\n",
    "    tf = [document.count(word) for word in intersection]\n",
    "    return np.array(tf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to initialize the counts of the shingles in order to capture the document frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize counts\n",
    "frequencies = {}\n",
    "text = open(\"data/portuguese/two_ends.txt\", \"r+\")\n",
    "for line in text:\n",
    "    word = line.strip().split(' ')\n",
    "    frequencies[word[0]] = float(word[1])   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finally putting it together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `score` is the combination of error modelling function and similarity function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(query, document, alpha = 0.5):\n",
    "    return alpha * pi(query, document) + (1.0 - alpha) * dirichlet(two_ends(query, 2), two_ends(document, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that our error modelling function is `pi` and we take `dirichlet` as the similarity function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pi(\"aspirat\", \"aspirar\"))\n",
    "print(dirichlet(two_ends(\"aspirat\", 2), two_ends(\"aspirar\", 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally you can see what score outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(score(\"aspirat\", \"aspirar\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, for every input cognate, we want to output a bunch of cognates which are similar to the other cognates in the list. This can be done in the following manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for source in data:\n",
    "    score_list = []\n",
    "    for key in wordlist:\n",
    "        score_list.append((key, score(key, source[0])))\n",
    "    score_list = sorted(score_list, key=lambda x: x[1], reverse=True)\n",
    "    \n",
    "    for i in range(5):\n",
    "        print(score_list[i])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}