sampling_demo.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from subsetCounting import SubsetSampling"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original experiment this code was designed for presented a participant with two stimuli to then make a judgement about. There were 34 stimuli divided in two groups.  Using all stimuli and presenting all pairings would require 34x34=1,056 pairwise comparisons (maintaining presentation order).  It was ultimately decided that using only one group of 17 would be sufficient with 17^2=289 pairwise comparisons, and not require an unreasonable number of participants.\n",
    "\n",
    "It was then decided that a subset of 5 stimuli would make a reasonably sized experiement for a single participant; 5x5=25 pairings with some number of repetitions.  The question then becomes: how many versions of the experiment will be needed to ensure that all 289 comparisons will be used?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The general process that `SubsetSampling` follows is to:\n",
    "1) create a 'pool' of elements to sample from, \n",
    "2) create lists of subsets to serve as mini 'experiments'\n",
    "3) count all all pairwise samples for each subset\n",
    "\n",
    "We would ideally like to have equal representation across all pairings.  This might be easy to accomplish for some pool sizes with particular subset sizes, but this is not always guarenteed to be possible (proof needed).  The best we can generally hope for is a fairly even distribution."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Manual Sampling"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can manually add subsets of your choosing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"poolSize\": 5,\n",
      "    \"subsetSize\": 2,\n",
      "    \"method\": \"\",\n",
      "    \"params\": {},\n",
      "    \"nSubsets\": 2,\n",
      "    \"min\": 0,\n",
      "    \"max\": 1\n",
      "}\n",
      " 0, 0, 0, 0, 0\n",
      " 0, 1, 1, 0, 0\n",
      " 0, 1, 2, 0, 1\n",
      " 0, 0, 0, 0, 0\n",
      " 0, 0, 1, 0, 1\n"
     ]
    }
   ],
   "source": [
    "p = SubsetSampling(pool_size=5, subset_size=2)\n",
    "samples = [[1,2],[2,4]]\n",
    "for s in samples:\n",
    "    p.updateObservationCounts(s, \"Add\")\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As subsets are added the self pairings along the diagonal will always be higher than pairs of other elements and are not considered for determining the maximum number of observed pairings shown in `info()`.\n",
    "\n",
    "You can make your own list of subsets to fill the `observationCounts` matrix to your own needs.  Keeping all values (excluding the diagonal) within a difference of 1 or 2 is harder than you may think!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Sampling"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Letting chance decide subsets for us can somewhat avoid the issue of over-represnentation.  Each subset can be generated with weights assigned to each index to help guide the pairwise comparisons to a more homogeneous state.\n",
    "\n",
    "The `buildRandomSubsets` function takes 2 parameters: The size of subset to make and the minumum number required for all pairs of comparisons.\n",
    "\n",
    "The `trim` function will attempt to remove subsets while still maintaining the minimum pairwise observations provided as a argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"poolSize\": 10,\n",
      "    \"subsetSize\": 5,\n",
      "    \"method\": \"generateRandomSubsets\",\n",
      "    \"params\": {\n",
      "        \"min_obs\": 1\n",
      "    },\n",
      "    \"nSubsets\": 8,\n",
      "    \"min\": 1,\n",
      "    \"max\": 4\n",
      "}\n",
      " 4, 2, 2, 1, 1, 4, 2, 2, 1, 1\n",
      " 2, 4, 3, 1, 2, 2, 2, 1, 2, 1\n",
      " 2, 3, 4, 1, 1, 2, 1, 2, 2, 2\n",
      " 1, 1, 1, 3, 1, 3, 1, 1, 2, 1\n",
      " 1, 2, 1, 1, 3, 2, 1, 1, 2, 1\n",
      " 4, 2, 2, 3, 2, 6, 3, 3, 3, 2\n",
      " 2, 2, 1, 1, 1, 3, 4, 2, 3, 1\n",
      " 2, 1, 2, 1, 1, 3, 2, 4, 3, 1\n",
      " 1, 2, 2, 2, 2, 3, 3, 3, 5, 2\n",
      " 1, 1, 2, 1, 1, 2, 1, 1, 2, 3\n"
     ]
    }
   ],
   "source": [
    "p = SubsetSampling(10, 5)\n",
    "p.generateRandomSubsets(1)\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the list of subsets that generate the above ObservationMatrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[8, 5, 3, 6, 7],\n",
       " [3, 1, 5, 0, 2],\n",
       " [8, 6, 2, 9, 1],\n",
       " [9, 3, 4, 8, 5],\n",
       " [8, 6, 5, 7, 0],\n",
       " [2, 7, 1, 8, 4],\n",
       " [9, 5, 2, 7, 0],\n",
       " [6, 4, 5, 1, 0]]"
      ]
     },
     "execution_count": 250,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.subsets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `n_iters` parameter will find the best list of subsets in that many attemps of random generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"poolSize\": 10,\n",
      "    \"subsetSize\": 5,\n",
      "    \"method\": \"generateRandomSubsets\",\n",
      "    \"params\": {\n",
      "        \"min_obs\": 1\n",
      "    },\n",
      "    \"nSubsets\": 6,\n",
      "    \"min\": 1,\n",
      "    \"max\": 2\n",
      "}\n",
      " 3, 1, 1, 1, 2, 2, 1, 1, 2, 1\n",
      " 1, 3, 1, 2, 1, 1, 2, 1, 2, 1\n",
      " 1, 1, 3, 2, 1, 2, 1, 1, 1, 2\n",
      " 1, 2, 2, 3, 2, 1, 1, 1, 1, 1\n",
      " 2, 1, 1, 2, 3, 1, 1, 2, 1, 1\n",
      " 2, 1, 2, 1, 1, 3, 2, 1, 1, 1\n",
      " 1, 2, 1, 1, 1, 2, 3, 2, 1, 1\n",
      " 1, 1, 1, 1, 2, 1, 2, 3, 1, 2\n",
      " 2, 2, 1, 1, 1, 1, 1, 1, 3, 2\n",
      " 1, 1, 2, 1, 1, 1, 1, 2, 2, 3\n"
     ]
    }
   ],
   "source": [
    "p = SubsetSampling(10, 5)\n",
    "p.generateRandomSubsets(1, n_iters=1000)\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stepped Sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider all possible ways to sample 5 elements from a set of 17.  This is 17 choose 5, or 6188 and using every single one would evenly saturate the pairwise matrix manytimes over.  It turns out that jumping through this set at even intervals yields good results.\n",
    "\n",
    "The `generateSubsets` function has two parameters: an initial offset, and the number of jumps to make.  One would need to vary both of these to find an optimal list of subsets.\n",
    "\n",
    "Using `trim` might be fruitful as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"poolSize\": 17,\n",
      "    \"subsetSize\": 5,\n",
      "    \"method\": \"generateSteppedSubsets\",\n",
      "    \"params\": {\n",
      "        \"offset\": 10,\n",
      "        \"amount\": 42\n",
      "    },\n",
      "    \"nSubsets\": 42,\n",
      "    \"min\": 0,\n",
      "    \"max\": 8\n",
      "}\n",
      "13, 4, 3, 4, 3, 3, 4, 4, 3, 3, 4, 4, 3, 3, 4, 2, 1\n",
      " 4,13, 3, 4, 5, 2, 4, 3, 1, 3, 5, 3, 4, 2, 4, 2, 3\n",
      " 3, 3,12, 3, 6, 1, 3, 1, 4, 4, 2, 1, 5, 3, 4, 4, 1\n",
      " 4, 4, 3,13, 4, 3, 3, 4, 4, 3, 3, 2, 2, 3, 5, 2, 3\n",
      " 3, 5, 6, 4,14, 2, 5, 1, 2, 3, 4, 3, 6, 0, 8, 2, 2\n",
      " 3, 2, 1, 3, 2,11, 4, 3, 2, 2, 5, 2, 3, 3, 3, 4, 2\n",
      " 4, 4, 3, 3, 5, 4,14, 4, 4, 2, 3, 2, 6, 1, 4, 5, 2\n",
      " 4, 3, 1, 4, 1, 3, 4,11, 4, 2, 4, 0, 2, 3, 3, 2, 4\n",
      " 3, 1, 4, 4, 2, 2, 4, 4,12, 3, 3, 2, 5, 3, 0, 5, 3\n",
      " 3, 3, 4, 3, 3, 2, 2, 2, 3,12, 4, 2, 5, 4, 4, 2, 2\n",
      " 4, 5, 2, 3, 4, 5, 3, 4, 3, 4,14, 2, 5, 2, 2, 6, 2\n",
      " 4, 3, 1, 2, 3, 2, 2, 0, 2, 2, 2, 8, 1, 3, 4, 0, 1\n",
      " 3, 4, 5, 2, 6, 3, 6, 2, 5, 5, 5, 1,15, 1, 6, 4, 2\n",
      " 3, 2, 3, 3, 0, 3, 1, 3, 3, 4, 2, 3, 1,10, 4, 2, 3\n",
      " 4, 4, 4, 5, 8, 3, 4, 3, 0, 4, 2, 4, 6, 4,16, 3, 6\n",
      " 2, 2, 4, 2, 2, 4, 5, 2, 5, 2, 6, 0, 4, 2, 3,12, 3\n",
      " 1, 3, 1, 3, 2, 2, 2, 4, 3, 2, 2, 1, 2, 3, 6, 3,10\n"
     ]
    }
   ],
   "source": [
    "p = SubsetSampling(17, 5)\n",
    "p.generateSteppedSubsets(10, 42)\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Programmatically trying many values of `offset` and `amount` may find a viable list of subsets depending on ones needs."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another method one can take is to completely take every possible subset and trim down to the desired minumum value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"poolSize\": 17,\n",
      "    \"subsetSize\": 5,\n",
      "    \"method\": \"generateSteppedSubsets\",\n",
      "    \"params\": {\n",
      "        \"offset\": 10,\n",
      "        \"amount\": 6188\n",
      "    },\n",
      "    \"nSubsets\": 6188,\n",
      "    \"min\": 455,\n",
      "    \"max\": 455\n",
      "}\n",
      "1820,455,455,455,455,455,455,455,455,455,455,455,455,455,455,455,455\n",
      "455,1820,455,455,455,455,455,455,455,455,455,455,455,455,455,455,455\n",
      "455,455,1820,455,455,455,455,455,455,455,455,455,455,455,455,455,455\n",
      "455,455,455,1820,455,455,455,455,455,455,455,455,455,455,455,455,455\n",
      "455,455,455,455,1820,455,455,455,455,455,455,455,455,455,455,455,455\n",
      "455,455,455,455,455,1820,455,455,455,455,455,455,455,455,455,455,455\n",
      "455,455,455,455,455,455,1820,455,455,455,455,455,455,455,455,455,455\n",
      "455,455,455,455,455,455,455,1820,455,455,455,455,455,455,455,455,455\n",
      "455,455,455,455,455,455,455,455,1820,455,455,455,455,455,455,455,455\n",
      "455,455,455,455,455,455,455,455,455,1820,455,455,455,455,455,455,455\n",
      "455,455,455,455,455,455,455,455,455,455,1820,455,455,455,455,455,455\n",
      "455,455,455,455,455,455,455,455,455,455,455,1820,455,455,455,455,455\n",
      "455,455,455,455,455,455,455,455,455,455,455,455,1820,455,455,455,455\n",
      "455,455,455,455,455,455,455,455,455,455,455,455,455,1820,455,455,455\n",
      "455,455,455,455,455,455,455,455,455,455,455,455,455,455,1820,455,455\n",
      "455,455,455,455,455,455,455,455,455,455,455,455,455,455,455,1820,455\n",
      "455,455,455,455,455,455,455,455,455,455,455,455,455,455,455,455,1820\n"
     ]
    }
   ],
   "source": [
    "p = SubsetSampling(17, 5)\n",
    "p.generateSteppedSubsets(10, p.totPoolSubsets)\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Warning! This can take a minute or two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"poolSize\": 17,\n",
      "    \"subsetSize\": 5,\n",
      "    \"method\": \"generateSteppedSubsets\",\n",
      "    \"params\": {\n",
      "        \"offset\": 10,\n",
      "        \"amount\": 6188\n",
      "    },\n",
      "    \"nSubsets\": 48,\n",
      "    \"min\": 2,\n",
      "    \"max\": 12\n",
      "}\n",
      "14, 2, 2, 2, 2, 2, 4, 2, 2, 2, 2, 2, 2, 3, 6, 9,12\n",
      " 2,13, 2, 2, 2, 3, 2, 3, 3, 3, 2, 2, 2, 2, 4, 7,11\n",
      " 2, 2,11, 2, 2, 3, 2, 3, 2, 3, 3, 2, 3, 2, 3, 3, 7\n",
      " 2, 2, 2,14, 2, 6, 2, 2, 4, 5, 4, 4, 2, 5, 3, 6, 5\n",
      " 2, 2, 2, 2,10, 2, 3, 2, 2, 4, 2, 2, 2, 3, 3, 3, 4\n",
      " 2, 3, 3, 6, 2,14, 2, 2, 2, 7, 6, 5, 3, 2, 2, 3, 6\n",
      " 4, 2, 2, 2, 3, 2,14, 5, 3, 6, 3, 7, 2, 2, 3, 4, 6\n",
      " 2, 3, 3, 2, 2, 2, 5,12, 3, 2, 3, 4, 4, 2, 2, 3, 6\n",
      " 2, 3, 2, 4, 2, 2, 3, 3,12, 2, 7, 4, 2, 3, 2, 4, 3\n",
      " 2, 3, 3, 5, 4, 7, 6, 2, 2,16, 5, 9, 2, 4, 3, 3, 4\n",
      " 2, 2, 3, 4, 2, 6, 3, 3, 7, 5,15, 5, 4, 5, 3, 2, 4\n",
      " 2, 2, 2, 4, 2, 5, 7, 4, 4, 9, 5,15, 3, 4, 2, 3, 2\n",
      " 2, 2, 3, 2, 2, 3, 2, 4, 2, 2, 4, 3,11, 2, 4, 2, 5\n",
      " 3, 2, 2, 5, 3, 2, 2, 2, 3, 4, 5, 4, 2,12, 2, 3, 4\n",
      " 6, 4, 3, 3, 3, 2, 3, 2, 2, 3, 3, 2, 4, 2,14, 5, 9\n",
      " 9, 7, 3, 6, 3, 3, 4, 3, 4, 3, 2, 3, 2, 3, 5,18,12\n",
      "12,11, 7, 5, 4, 6, 6, 6, 3, 4, 4, 2, 5, 4, 9,12,25\n"
     ]
    }
   ],
   "source": [
    "p.trim(2)\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the `trim` method is not perfect and could use some improvement!  However, trying lots of values for parameters, or multiple random generation attempts can work."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Store Parameters"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is one way to generate many subsets to evaluate based on ones needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tot 70, step 17.5\n",
      "[(0, 1, 2, 3), (0, 2, 3, 7), (1, 2, 3, 4), (1, 4, 6, 7)]\n",
      "{\n",
      "    \"poolSize\": 8,\n",
      "    \"subsetSize\": 4,\n",
      "    \"method\": \"generateSteppedSubsets\",\n",
      "    \"params\": {\n",
      "        \"offset\": 2,\n",
      "        \"amount\": 4\n",
      "    },\n",
      "    \"nSubsets\": 4,\n",
      "    \"min\": 0,\n",
      "    \"max\": 3\n",
      "}\n",
      " 2, 1, 2, 2, 0, 0, 0, 1\n",
      " 1, 3, 2, 2, 2, 0, 1, 1\n",
      " 2, 2, 3, 3, 1, 0, 0, 1\n",
      " 2, 2, 3, 3, 1, 0, 0, 1\n",
      " 0, 2, 1, 1, 2, 0, 1, 1\n",
      " 0, 0, 0, 0, 0, 0, 0, 0\n",
      " 0, 1, 0, 0, 1, 0, 1, 1\n",
      " 1, 1, 1, 1, 1, 0, 1, 2\n"
     ]
    }
   ],
   "source": [
    "params = {\"offset\": 2, \"amount\":4}\n",
    "p = SubsetSampling(8, 4)\n",
    "p.generateSteppedSubsets(**params)\n",
    "p.info()\n",
    "p.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SubsetSampling` class will store information that can be used to evaluate how good of a subset list it generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'poolSize': 8,\n",
       " 'subsetSize': 4,\n",
       " 'method': 'generateSteppedSubsets',\n",
       " 'params': {'offset': 2, 'amount': 4},\n",
       " 'nSubsets': 4,\n",
       " 'min': 0,\n",
       " 'max': 3}"
      ]
     },
     "execution_count": 264,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.params"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}