evaluation/Rank ordering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rank order\n",
    "\n",
    "Given a prediction $f$ and grid locations $(g_i)_{i=1}^N$ convert $f$ into \"percentiles\":\n",
    "$$ p(g) = \\frac{1}{N} \\sum_{i=1}^N [f(g) \\geq f(g_i)] $$\n",
    "where $[\\cdot]$ is the [Iverson bracket](https://en.wikipedia.org/wiki/Iverson_bracket), and $g$ is a grid location.  This process has an obviously close link with picking a coverage level.\n",
    "\n",
    "If we then have actual events $(x_i)_{i=1}^n$ we convert these into a series of values $(a_i)_{i=1}^n \\subseteq [0,1]$ by\n",
    "$$ a_i = p(x_i) $$\n",
    "\n",
    "We now proceed to work with the series $(a_i)$.  Source [1] seems to\n",
    "\n",
    "1. Look at the mean $\\mu = \\frac1n \\sum_{i=1}^n a_i$\n",
    "\n",
    "2. If we have two predictions $f_1,f_2$ form $p_1,p_2$ and hence series $(a_i^{(1)})$ and $(a_i^{(2)})$.  Then compare these, e.g. by comparing the means, or looking at\n",
    "$$ \\delta = \\frac1n \\sum_{i=1}^n [a_i^{(1)} > a_i^{(2)}]. $$\n",
    "\n",
    "Paper [1] works with these ideas in a hypothesis testing framework, but it is not clear to us if the used z-test is appropriate, given the likely correlation between the different $a_i$.\n",
    "\n",
    "\n",
    "## Literature\n",
    "\n",
    "1. Donald E. Brown, Hua Liu and Yifie Xue, \"Mining Preferences from Spatial-Temporal Data\".  Paper at [SDM'2001](http://www.siam.org/meetings/sdm01/) available [here](http://www.siam.org/meetings/sdm01/html/program.htm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advantages\n",
    "\n",
    "- Easy to compute\n",
    "- Depends only on the relative risk, and hence might be better suited to some of the \"a-theoretical\" prediction methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Disadvantages\n",
    "\n",
    "- Not really based on a probability model, so hard to be precise about what the measures actually show.\n",
    "- As we work explicitly with a grid, some adjustment would be need to work with a network where the _length_ of edges could vary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}