0_Intro_to_python_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to working with text data in Python\n",
    "\n",
    "## Contents\n",
    "1. [Setup](#Section-1%3A-Setup)\n",
    "    1. [Load python libraries](#1.1-Imports)\n",
    "    2. [Downloading Data](#1.2-Downloading-Data)\n",
    "    3. [Reading Data](#1.3-Reading-Data)\n",
    "    4. [Getting a Feel for Data](#1.2-Getting-a-Feel-for-Data)\n",
    "1. [Handling Text in Python](#Section-2%3A-Handling-Text-in-Python)\n",
    "    1. [Printing Text](#2.1-Printing-Text)\n",
    "    1. [Splitting and Counting Text](#2.2-Splitting-and-Counting-Text)\n",
    "    1. [Downloading Data](#1.2-Downloading-Data)\n",
    "1. [Functions](#Section-3%3A-Functions)\n",
    "    1. [Defining Our Own Functions](#3.1-Defining-Our-Own-Functions)\n",
    "1. [Text from Data](#Section-4%3A-Text-from-Data)\n",
    "1. [Pandas apply()](#Section-5%3A-Pandas-apply-Function)\n",
    "1. [Errors](#Section-6%3A-Errors)\n",
    "    1. [Understanding the Error](#6.1-Understanding-the-Error)\n",
    "    1. [Fixing the Error](#6.2-Fixing-the-Error)\n",
    "1. [Results as Data](#Section-7%3A-Results-as-Data)\n",
    "\n",
    "## Section 0: Background\n",
    "### 0.1 Learning Objectives\n",
    "- Be able to read data into python and save it back to files\n",
    "- Be able to view data and basic summary statistics using pandas\n",
    "- Learn to manipulate text strings\n",
    "- Learn how to use pandas' powerful .apply() function\n",
    "\n",
    "### 0.2 Data\n",
    "- We'll be using some profile data that OKCupid has made publically available.\n",
    "- The data is anonymized.\n",
    "- The data includes:\n",
    "    - Demographic information like age, gender, race/ethnicity, height, and income\n",
    "    - Profile text (what people wrote about themselves)    \n",
    "    \n",
    "@Author: [Jeff Lockhart](http://www-personal.umich.edu/~jwlock/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Setup\n",
    "### 1.1 Imports\n",
    "The first thing we need to do is import the tools we will be using.\n",
    "- Pandas is a \"package\" that contains python tools (in programming terms: \"functions\" and \"data structures\") for working with data. We'll see some of these tools in action in this lab. \n",
    "- The first part of the first line, `import pandas`, does exactly what it says. It brings the pandas tools into our current notebook so that we can use them.\n",
    "- The second part, `as pd`, lets us refer to pandas by typing `pd` instead of `pandas`. This shortcut is not required, but it is very common and popular. Programmers often like to cut down on typing by making names very short. \n",
    "- When you run this code, it will import pandas, but it won't show us any output (unless there is an error), because we have not asked it to show us any output. Computers are very literal: they do exactly and only what we tell them to.\n",
    "- There are several ways to run the code in a \"cell\" (those grey boxes with code in them). First, click anywhere inside the cell. Then you can either click the ▶ button near the top left of the page, or you can press [shift]+[enter] to run that cell and jump to the next one. \n",
    "    - Before a cell has been run, it will say `In [ ]:` on the left, indicating that this is a cell where you **in**put code.\n",
    "    - While the code in a cell is running, that is, when the computer is doing what the code tells it to, it will say `In [*]:` on the left.\n",
    "    - After the code in a cell has finished running, it will say something like `In [1]:` or `In [2]:`. The number is the order in which that code ran. So 1 was run first, then 2, and so on. If you run a cell a second time, its number will update to the latest number. \n",
    "    - If the code produces some output for the user to see, below the cell a new box will be created and labeled `Out [x]:`, where `x` is the number of the input cell that generated it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We have to install the gdown package using pip, a python package manager. \n",
    "# This allows the \"import gdown\" in next step. Other packages have been pre-installed in Python.\n",
    "!pip install gdown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import urllib.request\n",
    "import os.path\n",
    "import gdown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Downloading Data\n",
    "The next thing to do is download some data\n",
    "This code does a lot, but the idea is straightforward. We'll explain it briefly here, but don't worry if it's still a little confusing. The rest of this notebook will move slower. \n",
    "- Notice that the first line starts with `if` and the fourth line starts with `else`. Here we're telling the computer that if something is true, then it should do the code indented below the `if` line. Or `else` (i.e. if it is not true), then do the code indented under the `else` line. It will either run the code under `if` or the code under `else`, but not both. \n",
    "- `os.path.isfile(...)` checks whether there is a file with the name we give it. If there is, it is `True`. Otherwise, it's `False`. \n",
    "- If the file is there already:\n",
    "    - The code prints a message saying that we already have the data. We simply write `print(...)` and it shows us whatever we put into the round brackets. When you run the cell, you will see this text pop up under it. \n",
    "- If it is not there already:\n",
    "    - The code prints a message saying that it is downloading the data file.\n",
    "    - Then it uses something from `urllib` that we imported earlier to download the data. In the round brackets, we tell is where the data is with `url=`, and we tell it what to save the data as with `filename=`.\n",
    "    - When it finishes, it prints a messae saying `Done!`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.isfile(\"data/profiles.csv.zip\"):\n",
    "    print(\"You already downloaded the data. Great!\")\n",
    "    \n",
    "else:\n",
    "    print('Downloading file...')\n",
    "    try:\n",
    "        gdown.download(id=\"1A9S9PrYe3jNKqqK2yyrUNA3aHzfAhQZS\",\n",
    "                        output=\"data/profiles.csv.zip\")\n",
    "    except Exception as e:\n",
    "        print ('Failed to download from primary Google Drive, trying the secondary location:'+ str(e))\n",
    "        try: \n",
    "            gdown.download(id=\"1nDxUpMeNZWikPdPGYG9jnybWKlZKYB7A\",\n",
    "                            output=\"data/profiles.csv.zip\")\n",
    "        except Exception as e:\n",
    "            print ('Failed to download from the secondary Google Drive location.')\n",
    "            print ('You can manually download the file in above URL using your web browser and put it under./data/ folder.')\n",
    "            print ('Or please ask GSI for help...')\n",
    "        else: \n",
    "            print ('Done!!')\n",
    "    else:    \n",
    "        print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Reading Data\n",
    "\n",
    "Let's use pandas to read in a data file.\n",
    "- `read_csv()` is one of the tools (functions) in pandas. We can give it the name of a csv file, and it will open that file, read the data in it, and bring that data into our notebook so we can work with it.\n",
    "- The way we get to functions within a package is by using the period, like this `pd.read_csv()`. That tells python \"go to `pd` (which we said earlier is short for `pandas`) and look inside it for something called `read_csv`.\"\n",
    "- `read_csv` needs to know *which* csv file we want to read. So we tell it we want the one in the `data` folder (aka directory) named `profiles.csv.zip` by typing `'data/profiles.csv.zip'` into the round brackets of `read_csv()`. All functions have round brackets, even if we don't have any information to give them. \n",
    "- Finally, we need to make a name for the data we just read, so that we can ask python to do things with it. I'll be simple and name it \"profiles\", but you could name it anything (including \"my_data\", \"geraffe\", \"anything\", etc.). It is good to pick descriptive names, so that you can easily remember what data the names refer to. \n",
    "- In programming, named data is called a \"variable,\" and in order to associate a variable / name with data, we \"assign\" the data to the variable by setting the variable (on the left) equal to the data (on the right). \n",
    "    - Note that the programming meaning of the word \"variable\" is different than the statistical meaning of \"variable.\" In programming, it's any piece of data that we have given a name to. (It \"varies\" because we could change the data associated with the name/variable.) In statistics, it often means information that we collect, such as age or income. More on this in a moment. \n",
    "- Again, there is no output because we have not asked for any."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles = pd.read_csv('data/profiles.csv.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Getting a Feel for Data\n",
    "Finally, some output to look at!\n",
    "- Our `csv` file is just a table of data, like you would see in excel. (You can even save from excel as a `csv` if you want to.)\n",
    "    - In statistics, we would call each column in this table a \"variable.\" \n",
    "- Pandas saved our data in something called a `DataFrame`, which is one of the tools (a data structure) in pandas. DataFrame has a number of tools that are useful for working with data tables. For instance, if we want to know how many rows and columns are in our data, we can use the `shape` tool, as in `profiles.shape`.\n",
    "- Like before, we need to give a name to the output of shape, i.e. we need to assign it to a variable so that we can do things with it in python. I have called it `s` for \"shape\", but it could be anything.\n",
    "- We can use the python function `print()` to show something on our screen. We want to see the number of rows and columns, so we can ask python to `print` the variable `s` that has that information.\n",
    "- We see that `s` is just a pair of numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = profiles.shape\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Understanding the output of shape\n",
    "- We can get just one number from `s` using what's called \"indexing.\" The numbers in `s` have an order, and we can ask for them based on their position in that order. \n",
    "- Python, and computers in general, count from 0 rather than from 1. So the first thing in `s` is actually \"number 0\" (or the \"zeroth element\"). This confuses many beginners, but it gets easier with practice.\n",
    "- If we want the number in position zero, we can type `s[0]`. \n",
    "    - Note that these are square brackets, not round brackets. Round brackets usually go with functions, while square brackets usually go with data and indices. Another way to think about the difference is that with round brackets you're giving information ('arguments') to a function, while with square brackets you're asking for some of the information from inside a variable.\n",
    "- `shape` always gives us the number of rows first, then the number of columns. (This order, rows then columns, is used almost everywhere that people use tables or matrices.) Thus, `s[0]` is the number of rows and `s[1]` is the number of columns.\n",
    "- We can ask the `print` function to print multiple things at once by passing it multiple arguments, each separated by a comma. `print` automatically puts a space between each thing we ask it to show us. If we want it to print text, we need to use quotation marks so that it knows those letters are text rather than the names of variables and functions in python. \n",
    "    - Python doesn't care whether we use 'single quotes' or \"double quotes\", but the open and close quotes do have to match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Our data have', s[0], 'rows and', s[1], \"columns.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Peaking at the data\n",
    "- There are a *lot* of rows in our data. Too many for our screen. Let's just peak at the top few rows using the `head()` function of pandas DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can also get the names of the columns in our data by asking for them.\n",
    "- Notice that we don't need to use the `print()` function here. Normally, if we want to show output, we need to tell python to show output (e.g. by using `print()`). However, Jupyter Notebooks are smart and super helpful. If the last line of code in a cell gives some result, and it is not assigned to a variable (i.e. if we didn't name that result something so we could refer to it later), then the notebook will show that result to us. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting columns\n",
    "- We can select a single column from our DataFrame the same way we selected a single number from the output of `shape`, by indexing. The columns have names instead of numbers, so we can index by name and ask for `profiles['age']` to get the column named \"age\". \n",
    "- Because there are many rows, we'll use the `head()` function again to show just the top few. Here, we tell it how many rows to show by passing that number as an argument. \n",
    "- Note that the rows are numbered, and the numbers start at 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['age'].head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic summary functions\n",
    "- Many basic statistics functions are built into pandas DataFrames. Thus, we can take the `age` column of `profiles` and calculate the mean (average) age using `.mean()`.\n",
    "- Sometimes we want a set of general summary statistics for a column of numbers. We can do that with pandas' `describe()` function.\n",
    "- Not all columns contain numbers. The `education` column has categories. We can list the categories and the number of profiles that selected each one with the `.value_counts()` function build into pandas DataFrames.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Mean age:\", profiles['age'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['height'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['education'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Handling Text in Python\n",
    "- Anything in quotes is treated as text in python. \n",
    "- Python stores text in a data type called `string`. (Internally, computers think about text as letters or characters that have all been strung together like beads on a bracelet.)\n",
    "- Characters are more than just letters and numbers. There are also things like spaces, tabs, punctuation, and what are called \"non-printing characters.\" For now, let's just look at the `\\n` in the code below. It was in some of the code above, too. Together, `\\` and `n` represent the invisible \"new line\" character. They tell python that we want a new line, the same as pressing [enter] on your keyboard starts a new line in an essay. Because we're programming what is in the text and computers are very literal, we want to see these otherwise invisible characters so we can specify exactly what we want and where. When python prints the output, it doesn't show us \"`\\n`\". Instead, it puts the text after that on a new line below the one before it. \n",
    "\n",
    "### 2.1 Printing Text\n",
    "- In the code below, we do the following:\n",
    "    - We enter some text within quotes and assign it to a variable named `t`.\n",
    "    - We print that text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = \"I like dogs, but I prefer my cat.\"\n",
    "print(t)\n",
    "print(\"text:\", t)\n",
    "print(\"text:\\n\", t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Splitting and Counting Text\n",
    "- We measure the length of this text with python's built in `len()` funciton. Because python thinks of text as a `string` of characters, the length of the text / string is the number of characters in it. `len()` is not a function of or inside anything else, so we don't need a period or anything before it. We tell it what we want to know the length of by giving the name of that variable as an argument inside its round brackets.\n",
    "- We use the `split()` function to split the string into words. `split()` is built into strings, so we use a period after the name of our string to call it. When we print the result, we see that it is a list of words. `split()` cut up the string into chunks based on where the spaces were.\n",
    "- We can split a string on other things, too. We just need to tell `split()` which character to split on by passing it as an argument in the round brackets. Here, we split the string into just two parts, based on where the commas are.\n",
    "- If we want to know how many words are in a string, we can use the `len()` function to count the words in the list of words we got using `split`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = len(t)\n",
    "print(\"Number of characters in text:\", c)\n",
    "\n",
    "w = t.split()\n",
    "print(\"Word list:\\n\", w)\n",
    "\n",
    "clauses = t.split(',')\n",
    "print(\"Clause list:\\n\", clauses)\n",
    "\n",
    "l = len(w)\n",
    "print(\"Number of words in text:\", l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In Fewer Words\n",
    "- We can even do all of the steps of a word count in a single line. Until now, we have been saving the output of functions as variables, and then giving those variables to other functions. It is easier to read and learn that way. Often, programmers like to make things more compact. In the next line of code, python does the following:\n",
    "    - Finds the result of `t.split()`\n",
    "    - Sends that result as an argument input to `len()`\n",
    "    - Prints both the string we give it and the result of `len()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of words in text:\", len(t.split()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: Functions\n",
    "- Python has many useful functions built in, and many more functions are available using packages like pandas. Sometimes, though, we want to make our own function. This is a simple example using code we have already seen. Let's write our own function to count the words in a string of text.\n",
    "\n",
    "### 3.1 Defining Our Own Functions\n",
    "- To define a new function in python:\n",
    "    - We start with the keyword `def` (short for \"define\").\n",
    "    - After `def`, we specify the name we want our function to have. In this case, that name is `word_count`, but it could be anything we want. (Note: function names cannot have spaces in them.)\n",
    "    - Our function needs to know what text it should count the words in. We need to give it an \"argument\". Thus, we add round brackets after the function name. Inside the round brackets, we pick the name we will give to the variable holding our string. Here it is called \"text\". \n",
    "    - At the end of this first line, we place a colon `:` to tell python that the code after this belongs inside the function. \n",
    "    - The lines after this are indented (the notebook should auto-indent for us) so that python knows they are within the function definition above them. If you have programmed in other languages before, this is probably different than you are used to. Python is a \"whitespace language\", which means that it uses the indentation of code to tell if it is part of a function. \n",
    "    - The next two lines are the same code as before: the function takes whatever string it will be given, calls it `text`, splits it into a list of words `w`, and measures the length of that list (counts the number of words) as `l`.\n",
    "    - The 4th line uses the keyword `return` to tell python that this function will give some information back when we use it. Here, we're giving back the word count.\n",
    "    - Functions \"return\" when they are done running. At that point, the computer \"returns\" to whichever part of the code was running before the function's name was called, and it brings back whatever the output (\"return value\") of the function was.  Code after this is not part of the function, and so it is not indented.\n",
    "- We use our new function just like we used other functions:\n",
    "    - We call our `word_count()` function, giving it the same string `t` as before, and save its output in the variable `n_words`. (Note that `t` is the name of the variable we gave to the function, and it can be different than `text`, which is what the function has nicknamed that variable internally.)\n",
    "    - We print the result list like before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_count(text):\n",
    "    w = text.split()\n",
    "    l = len(w)\n",
    "    return l\n",
    "\n",
    "t = \"We don't need no education\\nWe don't need no thought control\\nNo dark sarcasm in the classroom\\nTeachers leave them kids alone\\nHey!\\nTeacher, leave us kids alone!\\nAll in all you're just another brick in the wall\\nAll in all you're just another brick in the wall\"\n",
    "print(t)\n",
    "\n",
    "print(\"\\nNumber of words in text:\", word_count(t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In Fewer Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_count2(text):\n",
    "    return len(text.split())\n",
    "\n",
    "print(\"Number of words in text:\", word_count2(t))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4: Text from Data\n",
    "\n",
    "#### There is text in the data we loaded. Let's peak at the first few rows of it.\n",
    "- Like before, the rows are numbered on the left\n",
    "- Note that the text is cut off with an `...` at the end of each line. The `head()` function does that because it wants to show us just the first part of the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['essay0'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's look at a single person's \"about me\" text\n",
    "- This code goes to the `profiles` data and selects the `essay0` column in it like before. This time, thoughm instead of asking for a peak at the top of that column, we ask it to give us the thing at position `1`.\n",
    "- Note that it is a sting just like what we were working with before, only this time, we got it as part of our data set rather than typing it into python directly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['essay0'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can save that string as a variable and work with it\n",
    "- Here we save it as `tmp`, which is a common name in programming for \"temporary\" things that won't be used again later. \n",
    "- When we use the `print()` function, the `\\n` newline characters get displayed correctly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = profiles['essay0'][1]\n",
    "print(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can do all the things we normally do with strings, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(word_count(tmp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(word_count(profiles['essay0'][1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 5: Pandas apply Function\n",
    "What if we want word count for all the profiles, not just one?\n",
    "- We can use `apply()`!\n",
    "- This code takes the `profiles['essay0']` column we have been working with and applies our `word_count` function to each thing in that column. \n",
    "    - The argument to `apply()` is the name of the function we want it to use on every person's profile text. \n",
    "    - Note that `word_count` here doesn't have round brackets `()` after it. We don't have to tell it what to count the words in because apply does that for us. It runs our function over and over, each time with a different person's text as the input.\n",
    "\n",
    "## Section 6: Errors\n",
    "But wait! The code below gives us an error. Run it and see, then see how we fix it below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['essay0'].apply(word_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Understanding the Error\n",
    "- The bottom of our error message tells us what went wrong `AttributeError: 'float' object has no attribute 'split'`.\n",
    "- Above that, the error helpfully shows us where in the code the error happened.\n",
    "    - We can see the error happened in our `word_count` function.\n",
    "    - It happened with the line `w = text.split()`.\n",
    "- Thus, the `text` we gave to our function must be something called a `float`, and a `float` must not work with the `split()` function. \n",
    "- If you encountered an error like this on your own, this would be a good time to paste the last line of the error into google and see how other people have solved it. Because this is a lab, we explain it here:\n",
    "    - `float` is a programming word for a number. It's actually short for \"floating point number,\" because floats can have decimal points in them. E.g. `3.14` is a `float`.\n",
    "    - Because `float`s are numbers and not `string`s, we can't do the same things with them. For instance, it makes sense to multiply two numbers, but multiplying text would be silly. Here, the error is complaining because we tried to use the string function `split` on a number, and splitting a number into words doesn't make sense. \n",
    "    \n",
    "### 6.2 Fixing the Error (Data Types)\n",
    "- We can write a new  `word_count()` function that makes sure everything is a `string` before using `split()` with it.\n",
    "- We'll use python's built in function `str()` to turn whatever is in the variable `text` into a string. \n",
    "- Look at the examples below:\n",
    "    - We make a variable called `pi` and set it equal to `3.14`. This is a number with a decimal, so it should be a `float`.\n",
    "    - We ask python what type of variable `pi` is with the function `type()`, and it tells us the type is `float`, like we expected.\n",
    "    - We tell python to turn `pi` into a string with the line `pi = str(pi)`. This time it shows up with quote marks `''` around it to let us know it's a string.\n",
    "    - We ask python what type it is now, and python tells us `str` meaning \"string\"\n",
    "- We added this line to our `word_count3()` function and tried it again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pi = 3.14\n",
    "pi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(pi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pi = str(pi)\n",
    "pi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(pi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_count3(text):\n",
    "    text = str(text)\n",
    "    w = text.split()\n",
    "    l = len(w)\n",
    "    return l\n",
    "\n",
    "profiles['essay0'].apply(word_count3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 7: Results as Data\n",
    "It worked! The result of our word count function is now data that we can use.\n",
    "- We got the number of words in every profile by applying our new function.\n",
    "- We probably want to save those somewhere, though, so we can use them later.\n",
    "- The code creates a new column and calls it `essay0_length`. \n",
    "- It sets that column equal to the result of our `apply()` code, which is the same as before.\n",
    "- It shows the head of our data again.\n",
    "- Look at the data and scroll all the way to the right: the last column is the one we just created, and it has the word counts in it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['essay0_length'] = profiles['essay0'].apply(word_count3)\n",
    "profiles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### These results are data now too\n",
    "`essay0_length` is just a column of numbers. We can treat it like any other data and do things like add up all the words in all the profiles with `sum()` or summaize the numbers in the column with `describe()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['essay0_length'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles['essay0_length'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## That's all, folks!\n",
    "This lab is a short introduction to the basics of using python and how computers think about text. There is a lot more we could cover, but this is plenty to get you going with the other labs. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}