Evaluating_Word_Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Word Embeddings [Description]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The favored method to compute the performance of our models is that of analogy tasks. The benchmark files utilized were generated by forming combinations of each w1 and w2 words from each relation in every dialect CSV file. The benchmark file consists of analogies, where lines are 4-tuples of words, split into sections by “: SECTION NAME” lines. Gensim library was used to evaluate the models and the accuracy was reported for each section separately, including an aggregate summary at the end. The chosen methodology to solving analogies is the vector offset (3CossAdd), which falls under the pair-based methods for solving analogies. A proportional analogy holds between two word pairs: a:a* :: b:b* (a is to a* as b is to b*) For example, Tokyo is to Japan as Paris is to France. With the pair-based methods, given a:a* :: b:?, the task is to find b*.  As a means of truly capturing a model’s accuracy, the final aggregated report includes a top-1, top-5, and top-10 accuracy result for each section, in which the b* word is searched for in the model’s top-k most similar words. Furthermore, the accuracies are computed over dummy4unknown=true, which produce zero accuracies for 4-tuples with out-of-vocabulary words and dummy4unknown=false, in which tuples are skipped entirely and not used in the evaluation.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim Function Signature --> evaluate_word_analogies()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**evaluate_word_analogies(analogies, restrict_vocab=300000, case_insensitive=True, dummy4unknown=False)**\n",
    "*Compute performance of the model on an analogy test set.*\n",
    "\n",
    "    This is modern variant of accuracy(), see discussion on GitHub #1935.This method corresponds to the compute-accuracy script of the original C word2vec. See also Analogy (State of the art).\n",
    "\n",
    "**Parameters**:\t\n",
    "\n",
    "- **analogies (str)** – Path to file, where lines are 4-tuples of words, split into sections by “: SECTION NAME” lines. See gensim/test/test_data/questions-words.txt as example.\n",
    "\n",
    "- **restrict_vocab (int, optional)** – Ignore all 4-tuples containing a word not in the first restrict_vocab words. This may be meaningful if you’ve sorted the model vocabulary by descending frequency (which is standard in modern word embedding models).\n",
    "\n",
    "- **case_insensitive (bool, optional)** – If True - convert all words to their uppercase form before evaluating the performance. Useful to handle case-mismatch between training tokens and words in the test set. In case of multiple case variants of a single word, the vector for the first occurrence (also the most frequent if vocabulary is sorted) is taken.\n",
    "\n",
    "- **dummy4unknown (bool, optional)** – If True - produce zero accuracies for 4-tuples with out-of-vocabulary words. Otherwise, these tuples are skipped entirely and not used in the evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Libararies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import glob \n",
    "import pandas as pd \n",
    "import pickle\n",
    "from smart_open import open\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.models import KeyedVectors\n",
    "from gensim.models import keyedvectors\n",
    "import xlwt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining Constants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**MODEL_PATH** - Path to the pre-trained model\n",
    "\n",
    "**BENCHMARKS_PATH** – Path to file, where lines are 4-tuples of words, split into sections by “: SECTION NAME” lines. See gensim/test/test_data/questions-words.txt as example.\n",
    "\n",
    "**TOP_K** - The final aggregated report includes a top-1, top-5, and top-10 accuracy result for each section, in which the b word is searched for in the model’s top-k most similar words\n",
    "\n",
    "**DUMMY4UKNOWN**  (bool, optional) – If True - produce zero accuracies for 4-tuples with out-of-vocabulary words. Otherwise, these tuples are skipped entirely and not used in the evaluation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "MODEL_PATH = \"/Users/jaddoughman/Desktop/Habibi.model\"\n",
    "BENCHMARKS_PATH = \"benchmarks/*.txt\"\n",
    "TOP_K = [1,5,10]\n",
    "DUMMY4UKNOWN = [True,False]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation + Writing to XLS Workbook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# A logger file will be created as \"logger_{modelname}.txt\"\n",
    "logging.basicConfig(filename=\"logger_\"+MODEL_PATH.split(\"/\")[-1].split(\".\")[0], format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jaddoughman/opt/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    }
   ],
   "source": [
    "# Loading pre-trained model \n",
    "\n",
    "model = KeyedVectors.load_word2vec_format(MODEL_PATH)\n",
    "\n",
    "#model = Word2Vec.load(MODEL_PATH)\n",
    "\n",
    "# OR \n",
    "#model = KeyedVectors.load(MODEL_PATH)\n",
    "\n",
    "# OR \n",
    "#model = Word2Vec.load_word2vec_format(MODEL_PATH,binary=True)\n",
    "\n",
    "# OR \n",
    "#model = KeyedVectors.load_word2vec_format(fname=MODEL_PATH,fvocab=None,binary=True, encoding='utf-8', unicode_errors='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Habibi_DZ\n",
      "Top 1 ---> dummy4unknown=True\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jaddoughman/opt/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:50: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).\n"
     ]
    }
   ],
   "source": [
    "style0 = xlwt.easyxf('font: name Times New Roman, bold on')\n",
    "# Creating an excel workbook which will contains various spreadsheets, one for each dialect benchmark file\n",
    "wb = xlwt.Workbook()\n",
    "\n",
    "# Iterating over each benchmark file to compute accuracy\n",
    "for benchmark in glob.iglob(BENCHMARKS_PATH): \n",
    "    \n",
    "    # Creating a spreadsheet for every benchmark file\n",
    "    ws = wb.add_sheet(benchmark.split(\"/\")[-1].split(\".\")[0])\n",
    "\n",
    "########################### XLS Formatting ##############################################################\n",
    "\n",
    "    ws.write(0, 0, \"OOV Penalty\", style0)\n",
    "    for i in range(1,4):\n",
    "        ws.write(0, i, \"false\", style0)\n",
    "    for i in range(4,7):\n",
    "        ws.write(0, i, \"true\", style0)\n",
    "\n",
    "    ws.write(1, 0, \"Relation / Top-K\", style0)\n",
    "    \n",
    "    \n",
    "    ws.write(1, 1, \"Top-1\", style0)\n",
    "    ws.write(1, 2, \"Top-5\", style0)\n",
    "    ws.write(1, 3, \"Top-10\", style0)\n",
    "    ws.write(1, 4, \"Top-1\", style0)\n",
    "    ws.write(1, 5, \"Top-5\", style0)\n",
    "    ws.write(1, 6, \"Top-10\", style0)\n",
    "    \n",
    "    ws.write(2, 0, \"Double\", style0)\n",
    "    ws.write(3, 0, \"Plural\", style0)\n",
    "    ws.write(4, 0, \"Genitive Past Tense\", style0)\n",
    "    ws.write(5, 0, \"Opposite\", style0)\n",
    "    ws.write(6, 0, \"Comparative\", style0)\n",
    "    ws.write(7, 0, \"Man-Woman\", style0)\n",
    "    ws.write(8, 0, \"Total Accuracy\", style0)\n",
    "\n",
    "###########################################################################################################\n",
    "\n",
    "    n_false = 1\n",
    "    n_true = 4 \n",
    "    \n",
    "    # Computing accuracy over top-k most similar words\n",
    "    for k in TOP_K:\n",
    "        #model = keyedvectors.WordEmbeddingsKeyedVectors.load(MODEL_PATH)\n",
    "        print(MODEL_PATH.split(\"/\")[-1].split(\".\")[0] + \"_\" + benchmark.split(\"/\")[-1].split(\".\")[0])\n",
    "        \n",
    "        # Computing accuracy over top-k most similar words when penalizing and not penalizing out-of-vocabulary words\n",
    "        for dummy in DUMMY4UKNOWN:     \n",
    "            print(\"Top\",k,\"---> dummy4unknown=\"+str(dummy))\n",
    "            sections_accuracy = model.wv.evaluate_word_analogies(benchmark,topn=int(k),dummy4unknown=dummy)\n",
    "            print(sections_accuracy,end=\"\\n\\n\")\n",
    "            \n",
    "            if dummy==False:\n",
    "                ws.write(2,n_false,\"None\") if sections_accuracy['double']==None else ws.write(2,n_false, (sections_accuracy['double']*100))\n",
    "                ws.write(3,n_false,\"None\") if sections_accuracy['plural']==None else ws.write(3,n_false, (sections_accuracy['plural']*100))\n",
    "                ws.write(4,n_false,\"None\") if sections_accuracy['genitive_past_tense']==None else ws.write(4,n_false, (sections_accuracy['genitive_past_tense']*100))\n",
    "                ws.write(5,n_false,\"None\") if sections_accuracy['opposite']==None else ws.write(5,n_false, (sections_accuracy['opposite']*100))\n",
    "                ws.write(6,n_false,\"None\") if sections_accuracy['comparative']==None else ws.write(6,n_false, (sections_accuracy['comparative']*100))\n",
    "                ws.write(7,n_false,\"None\") if sections_accuracy['man_woman']==None else ws.write(7,n_false, (sections_accuracy['man_woman']*100))\n",
    "                ws.write(8,n_false,\"None\") if sections_accuracy['total']==None else ws.write(8,n_false, (sections_accuracy['total']*100))\n",
    "                n_false = n_false + 1\n",
    "                \n",
    "            if dummy==True:\n",
    "                ws.write(2,n_true,\"None\") if sections_accuracy['double']==None else ws.write(2,n_true, (sections_accuracy['double']*100))\n",
    "                ws.write(3,n_true,\"None\") if sections_accuracy['plural']==None else ws.write(3,n_true, (sections_accuracy['plural']*100))\n",
    "                ws.write(4,n_true,\"None\") if sections_accuracy['genitive_past_tense']==None else ws.write(4,n_true, (sections_accuracy['genitive_past_tense']*100))\n",
    "                ws.write(5,n_true,\"None\") if sections_accuracy['opposite']==None else ws.write(5,n_true, (sections_accuracy['opposite']*100))\n",
    "                ws.write(6,n_true,\"None\") if sections_accuracy['comparative']==None else ws.write(6,n_true, (sections_accuracy['comparative']*100))\n",
    "                ws.write(7,n_true,\"None\") if sections_accuracy['man_woman']==None else ws.write(7,n_true, (sections_accuracy['man_woman']*100))\n",
    "                ws.write(8,n_true,\"None\") if sections_accuracy['total']==None else ws.write(8,n_true, (sections_accuracy['total']*100))\n",
    "                n_true = n_true + 1        \n",
    "    print(50*\"*\")\n",
    "    \n",
    "# Saving workbook as : \"{modelname}.xls\"\n",
    "wb.save(MODEL_PATH.split(\"/\")[-1].split(\".\")[0]+\".xls\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}