HW4 - [YOUR NAME HERE] - 2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 4  \n",
    "# \"People like you...\"\n",
    "### Assigned Tuesday, 9 April 2019  /  Due Friday, 19 April 2018 at 11:59 pm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives:\n",
    "\n",
    "We can use cosine similarity to compare users to users or movies to movies. Here we're going to use cosine similarity and PCA to postulate relationships between individuals. In a very real sense, you're going to be engaged in some of the practices of data preparation common in machine learning research.  \n",
    "\n",
    "We're then going to reflect on the implications of what we did. \n",
    "\n",
    "More specifically, you're going to\n",
    "1. Get the \"Adult\" machine learning data set and convert nonnumerical information into numerical values;\n",
    "2. Employ cosine simularities to compare individuals;\n",
    "3. Use PCA to graphically represent this data in two dimensions;\n",
    "4. Use K-means on PCA results to generate clusters within data (in homework 5).\n",
    "5. Reflect upon the ethical implications of this approach and the construction of the data set (in homework 5). \n",
    "\n",
    "We'll build on what we do here in homework 5, where we'll take our PCA results and perform K-means on them to generate clusters. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions \n",
    "This assignment is to be done <b>on your own</b>, but you can talk about the assignment with your classmates if you get stuck. (Be sure to list the students you spoke with about this assignment in the space provided below.) Feel free to also use [stackoverflow](https://stackoverflow.com/) but please provide citation and link to the specific answer if you do this. You may also visit Will Yumou during his TA office hours or email him with questions.\n",
    "\n",
    "Provide your code to justify your answer to each question. \n",
    "\n",
    "Be sure to rename this homework notebook so that it includes your name. \n",
    "\n",
    "### List any students you talked with about this assignment here:\n",
    "1. [person 1]\n",
    "2. [person 2]\n",
    "3. etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework Problems \n",
    "####  Be sure to rename this homework so that it includes your name. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### question 1 [20 points] \n",
    "1. Download an \"adult\" data set from the machine learning repository (https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data) and import this into pandas dataframe named `data`. I've given you the column names in the list `features` below. Rename the columns using the `features` list. To help get you started, I've given you some libraries you'll need.\n",
    "2. To get a sense of the data and make sure you've named the columns properly, use the `head` command.\n",
    "3. <b>Using the `preprocess_features` function code provided</b>, convert strings values in  `data` set to numerical values. \n",
    "4. Use `head` again to confirm you successfully converted features into numerical values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Some code to get you started..\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "plt.style.use\n",
    "plt.rcParams['figure.figsize'] = (15, 5)\n",
    "\n",
    "# a list of column names to use in the dataframe `data`\n",
    "features = ['age','workclass', 'fnlwgt', 'education', 'education.num', 'marital.status','occupation', 'relationship','race','sex','capital.gain','capital.loss', 'hours.per.week', 'native.country','income']\n",
    "\n",
    "# Provide your code for steps 1 and 2 of Q1 here:\n",
    "# ...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provided code for step 3 of Q1\n",
    "\n",
    "## converts strings values in data to numerical values\n",
    "## from https://www.kaggle.com/dewilliams/ml-adult-income\n",
    "\n",
    "# THE FOLLOWING CODE DEFINES A FUNCTION CALLED `preprocess_features`. \n",
    "# YOU CAN USE `preprocess_features` ON A DATAFRAME `data` LIKE THIS: data = preprocess_features(data)\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "def preprocess_features(dframe):\n",
    "    for column in dframe:\n",
    "        enc = LabelEncoder()\n",
    "        if(column not in ['age','education.num','fnlwgt','capital.gain','capital.loss','hours.per.week']):\n",
    "            dframe[column] = enc.fit_transform(dframe[column])\n",
    "    return dframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code for steps 3 and 4 of Q1 \n",
    "# ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### question 2 [10 points]\n",
    "Now that we've converted all the categorical data to numerical data, let's normalize all of these variables (i.e., columns) to be between 0.0 and 1.0. This is an interpretative decision: we are effectively deciding not to allow any of these variables to have a greater maximum or minimum magnitude than any other variable. (We might also want to argue about how we coverted our categorical variables into continous variables. For what kind of questions would this be a mistake?) <b>Yet for the purposes of this homework and demonstration, we press on.</b>    \n",
    "\n",
    "We can take advantage of pandas to normalize our dataframe 'data'. In the interests of saving you time, here's the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize dataframe \"data\"\n",
    "normalized_data = (data-data.min())/(data.max()-data.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, much like what we might do with films, we want to compute the cosine similarity between each person based on their attributes. Below I give you an example for producing cosine similarity scores for the first person in \"data\" and everyone else:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EXAMPLE CODE FOR COMPUTING COSINE SIMILARITY FOR ONE PERSON\n",
    "# If this step gives you a memory error or kernel crash, use normalized_data[0:5000] in place of normalized_data\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "similarity_for_one_person = cosine_similarity(normalized_data.iloc[[0]], normalized_data)\n",
    "similarity_for_one_person"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Given the above example, calculate the cosine similarity for the first 5000 people in the dataframe \"data\".</b> Name this array `similarity`. \n",
    "\n",
    "(NOTE: Similar to the situation for HW3, we're only doing the first 5000 people because the RAM limitations of Codio prevent us from doing more people.)\n",
    "\n",
    "Hint 1: The solution to this is just one line of code. If you want to take cosine similarity of a whole array (rather than an individual row as we did above), you can enter the array as the sole argument of the cosine_similarity function. You can check your answer by (1) comparing the first row to the answer above for one person.  \n",
    "Hint 2: If the step below gives you a memory error or kernel crash, use `normalized_data[0:2000]` in place of `normalized_data[0:5000]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here for question 2 here\n",
    "# ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### question 3 [30 points] \n",
    "Now we have a similarity matrix describing a set of relationships postulated between different individuals in the data set based on the similarity of the 15 features in the data set. We'd like to visualize this similarity to get some sense of how the individuals are distributed with respect to each other. To do this, we again return to our trusty unsupervised learning technique: PCA. \n",
    "\n",
    "For this question, make PCA plots for three different sets of people from `similarity`: \n",
    "1. PCA on the <b>first 100</b> individuals in `similarity`. \n",
    "2. PCA on the <b>first 1000</b> individuals in `similarity`. \n",
    "3. PCA on the <b>last 1000</b> individuals in `similarity`. \n",
    "\n",
    "\n",
    "<b>NOTE 1: WE WOULD NORMALLY USE PCA ON THE ENTIRE DATA SET OF 32500 PEOPLE, BUT, JUST LIKE FOR HOMEWORK 3, CODIO DOES NOT PROVIDE YOU WITH ENOUGH RAM TO PLOT THE FULL DATA SET.</b> AARON WILL SHOW YOU A PCA PLOT OF THE FULL DATA SET DURING LAB.\n",
    "\n",
    "NOTE 2: If you get a memory error/kernel crash, run the PCA above for fewer people, e.g., 300 instead of 1000, etc. \n",
    "\n",
    "We'll reflect deeply on the ethics of using this data set and on performing similarity measures on *people* in our next and final homework, using the work we did here as our starting point. For purposes of time we'll end this homework here with the caveat that there remains much to discuss and that we need to continue this conversation in part II of this homework (i.e., homework 5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# libraries to get you started\n",
    "from sklearn.decomposition import PCA as PCA \n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your PCA code here for the first 100 individuals here\n",
    "# ...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Enter your PCA code here for the first 1000 individuals here\n",
    "# ...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your PCA code here for the last 1000 individuals here\n",
    "# ...\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}