character-encodings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76e87862",
   "metadata": {
    "papermill": {
     "duration": 0.006019,
     "end_time": "2022-10-12T06:53:28.605478",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.599459",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Data Cleaning](https://www.kaggle.com/learn/data-cleaning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/character-encodings).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d2ea855",
   "metadata": {
    "papermill": {
     "duration": 0.004772,
     "end_time": "2022-10-12T06:53:28.615420",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.610648",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you'll apply what you learned in the **Character encodings** tutorial.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8b012209",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.627890Z",
     "iopub.status.busy": "2022-10-12T06:53:28.626963Z",
     "iopub.status.idle": "2022-10-12T06:53:28.724941Z",
     "shell.execute_reply": "2022-10-12T06:53:28.723667Z"
    },
    "papermill": {
     "duration": 0.108051,
     "end_time": "2022-10-12T06:53:28.728353",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.620302",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.data_cleaning.ex4 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "821df5f3",
   "metadata": {
    "papermill": {
     "duration": 0.004607,
     "end_time": "2022-10-12T06:53:28.738523",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.733916",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Get our environment set up\n",
    "\n",
    "The first thing we'll need to do is load in the libraries we'll be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c1a733fe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.750761Z",
     "iopub.status.busy": "2022-10-12T06:53:28.750359Z",
     "iopub.status.idle": "2022-10-12T06:53:28.755522Z",
     "shell.execute_reply": "2022-10-12T06:53:28.754597Z"
    },
    "papermill": {
     "duration": 0.013525,
     "end_time": "2022-10-12T06:53:28.757699",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.744174",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# helpful character encoding module\n",
    "import chardet\n",
    "\n",
    "# set seed for reproducibility\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa3d3b90",
   "metadata": {
    "papermill": {
     "duration": 0.004941,
     "end_time": "2022-10-12T06:53:28.767710",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.762769",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1) What are encodings?\n",
    "\n",
    "You're working with a dataset composed of bytes.  Run the code cell below to print a sample entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "99f375c8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.779382Z",
     "iopub.status.busy": "2022-10-12T06:53:28.778447Z",
     "iopub.status.idle": "2022-10-12T06:53:28.784155Z",
     "shell.execute_reply": "2022-10-12T06:53:28.782822Z"
    },
    "papermill": {
     "duration": 0.014172,
     "end_time": "2022-10-12T06:53:28.786639",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.772467",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\xa7A\\xa6n'\n",
      "data type: <class 'bytes'>\n"
     ]
    }
   ],
   "source": [
    "sample_entry = b'\\xa7A\\xa6n'\n",
    "print(sample_entry)\n",
    "print('data type:', type(sample_entry))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72dc5dbd",
   "metadata": {
    "papermill": {
     "duration": 0.005121,
     "end_time": "2022-10-12T06:53:28.796752",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.791631",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You notice that it doesn't use the standard UTF-8 encoding. \n",
    "\n",
    "Use the next code cell to create a variable `new_entry` that changes the encoding from `\"big5-tw\"` to `\"utf-8\"`.  `new_entry` should have the bytes datatype."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "396ec2c4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.809739Z",
     "iopub.status.busy": "2022-10-12T06:53:28.808737Z",
     "iopub.status.idle": "2022-10-12T06:53:28.821109Z",
     "shell.execute_reply": "2022-10-12T06:53:28.819752Z"
    },
    "papermill": {
     "duration": 0.02124,
     "end_time": "2022-10-12T06:53:28.823704",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.802464",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.3333333333333333, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"1_EncodingsIntro\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "before = sample_entry.decode(\"big5-tw\")\n",
    "new_entry = before.encode()\n",
    "\n",
    "# Check your answer\n",
    "q1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "59cdd5ef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.836504Z",
     "iopub.status.busy": "2022-10-12T06:53:28.835544Z",
     "iopub.status.idle": "2022-10-12T06:53:28.840977Z",
     "shell.execute_reply": "2022-10-12T06:53:28.839232Z"
    },
    "papermill": {
     "duration": 0.014334,
     "end_time": "2022-10-12T06:53:28.843391",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.829057",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q1.hint()\n",
    "#q1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1ed01b3",
   "metadata": {
    "papermill": {
     "duration": 0.004972,
     "end_time": "2022-10-12T06:53:28.853538",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.848566",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2) Reading in files with encoding problems\n",
    "\n",
    "Use the code cell below to read in this file at path `\"../input/fatal-police-shootings-in-the-us/PoliceKillingsUS.csv\"`.  \n",
    "\n",
    "Figure out what the correct encoding should be and read in the file to a DataFrame `police_killings`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1eb84392",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.866546Z",
     "iopub.status.busy": "2022-10-12T06:53:28.865764Z",
     "iopub.status.idle": "2022-10-12T06:53:28.884989Z",
     "shell.execute_reply": "2022-10-12T06:53:28.883682Z"
    },
    "papermill": {
     "duration": 0.029068,
     "end_time": "2022-10-12T06:53:28.888204",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.859136",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: Load in the DataFrame correctly.\n",
    "police_killings = pd.read_csv(\"../input/fatal-police-shootings-in-the-us/PoliceKillingsUS.csv\", encoding='Windows-1252')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2ba64f1",
   "metadata": {
    "papermill": {
     "duration": 0.005628,
     "end_time": "2022-10-12T06:53:28.899083",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.893455",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Feel free to use any additional code cells for supplemental work.  To get credit for finishing this question, you'll need to run `q2.check()` and get a result of **Correct**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c38871d4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.913093Z",
     "iopub.status.busy": "2022-10-12T06:53:28.912664Z",
     "iopub.status.idle": "2022-10-12T06:53:28.917869Z",
     "shell.execute_reply": "2022-10-12T06:53:28.916586Z"
    },
    "papermill": {
     "duration": 0.014133,
     "end_time": "2022-10-12T06:53:28.920110",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.905977",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# (Optional) Use this code cell for any additional work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d06cd801",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.933416Z",
     "iopub.status.busy": "2022-10-12T06:53:28.932937Z",
     "iopub.status.idle": "2022-10-12T06:53:28.937743Z",
     "shell.execute_reply": "2022-10-12T06:53:28.936584Z"
    },
    "papermill": {
     "duration": 0.013706,
     "end_time": "2022-10-12T06:53:28.940228",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.926522",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q2.hint()\n",
    "#q2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccbc89ce",
   "metadata": {
    "papermill": {
     "duration": 0.004806,
     "end_time": "2022-10-12T06:53:28.950263",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.945457",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Saving your files with UTF-8 encoding\n",
    "\n",
    "Save a version of the police killings dataset to CSV with UTF-8 encoding.  Your answer will be marked correct after saving this file.  \n",
    "\n",
    "Note: When using the `to_csv()` method, supply only the name of the file (e.g., `\"my_file.csv\"`).  This saves the file at the filepath `\"/kaggle/working/my_file.csv\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "48dc555c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:28.963291Z",
     "iopub.status.busy": "2022-10-12T06:53:28.962645Z",
     "iopub.status.idle": "2022-10-12T06:53:28.985065Z",
     "shell.execute_reply": "2022-10-12T06:53:28.983750Z"
    },
    "papermill": {
     "duration": 0.03197,
     "end_time": "2022-10-12T06:53:28.987846",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.955876",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: Save the police killings dataset to CSV\n",
    "police_killings.to_csv(\"my_file.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "14e0da13",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-12T06:53:29.000415Z",
     "iopub.status.busy": "2022-10-12T06:53:28.999916Z",
     "iopub.status.idle": "2022-10-12T06:53:29.004644Z",
     "shell.execute_reply": "2022-10-12T06:53:29.003430Z"
    },
    "papermill": {
     "duration": 0.014098,
     "end_time": "2022-10-12T06:53:29.007235",
     "exception": false,
     "start_time": "2022-10-12T06:53:28.993137",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q3.hint()\n",
    "#q3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e1e1fef",
   "metadata": {
    "papermill": {
     "duration": 0.004841,
     "end_time": "2022-10-12T06:53:29.017403",
     "exception": false,
     "start_time": "2022-10-12T06:53:29.012562",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# (Optional) More practice\n",
    "\n",
    "Check out [this dataset of files in different character encodings](https://www.kaggle.com/rtatman/character-encoding-examples). Can you read in all the files with their original encodings and them save them out as UTF-8 files?\n",
    "\n",
    "If you have a file that's in UTF-8 but has just a couple of weird-looking characters in it, you can try out the [ftfy module](https://ftfy.readthedocs.io/en/latest/#) and see if it helps. \n",
    "\n",
    "# Keep going\n",
    "\n",
    "In the final lesson, learn how to [**clean up inconsistent text entries**](https://www.kaggle.com/alexisbcook/inconsistent-data-entry) in your dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc0972a1",
   "metadata": {
    "papermill": {
     "duration": 0.004787,
     "end_time": "2022-10-12T06:53:29.027696",
     "exception": false,
     "start_time": "2022-10-12T06:53:29.022909",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/data-cleaning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 10.572925,
   "end_time": "2022-10-12T06:53:29.655176",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-10-12T06:53:19.082251",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}