2. preprocessing raw data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing Raw Data\n",
    "\n",
    "Here will do some simple preprocessing steps on raw data in order to make sure it's ready for the feature engineering phase.\n",
    "\n",
    "## 1. Balance of target classes\n",
    "\n",
    "In the first step, we have to make sure if the raw data is balanced. An unbalanced data would sure affect the predictive model and lead to a biased result. Thus, we'll check whether class distribution in each dataset is balanced. If not, then we can carry out necessary steps, such as oversampling or undersampling, after feature engineering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# import our modules here\n",
    "from modules.DataHandler import *\n",
    "from modules.utils import *\n",
    "\n",
    "# HandleData: used for downloading dataset (txt) files and handle the data we get\n",
    "datahandler = DataHandler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=datahandler.load_txt('UCI HAR Dataset/train/subject_train.txt')\n",
    "class_count = pd.value_counts(data.iloc[:,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "TRAIN-SET\n",
      "class\tnumber\t%-wise\n",
      "6\t1407\t19.14\n",
      "5\t1374\t18.69\n",
      "4\t1286\t17.49\n",
      "1\t1226\t16.68\n",
      "2\t1073\t14.59\n",
      "3\t986\t13.41\n",
      "\n",
      "TEST-SET\n",
      "class\tnumber\t%-wise\n",
      "6\t537\t18.22\n",
      "5\t532\t18.05\n",
      "1\t496\t16.83\n",
      "4\t491\t16.66\n",
      "2\t471\t15.98\n",
      "3\t420\t14.25\n"
     ]
    }
   ],
   "source": [
    "prefix = ['train', 'test']\n",
    "\n",
    "for p in prefix:\n",
    "    print('\\n{}-SET\\nclass\\tnumber\\t%-wise'.format(p.upper()))\n",
    "    \n",
    "    # load file\n",
    "    data=datahandler.load_txt('UCI HAR Dataset/{p}/y_{p}.txt'.format(p=p))\n",
    "    # count the number of occurance for each classes\n",
    "    class_count = pd.value_counts(data.iloc[:,0])\n",
    "    \n",
    "    for c in range(len(class_count)):\n",
    "        print('%d\\t%d\\t%.2f' %(class_count.index[c], \n",
    "                               class_count.values[c], \n",
    "                               100*class_count.values[c]/sum(class_count)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*It looks like distribution of our clsses is fairly normal. So it's safe to continue.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Missing value\n",
    "\n",
    "Check if there's any missing value on every raw data file. If so, remove/replace it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "TRAIN\n",
      "body_acc_x_train: No missing value\n",
      "body_acc_y_train: No missing value\n",
      "body_acc_z_train: No missing value\n",
      "body_gyro_x_train: No missing value\n",
      "body_gyro_y_train: No missing value\n",
      "body_gyro_z_train: No missing value\n",
      "total_acc_x_train: No missing value\n",
      "total_acc_y_train: No missing value\n",
      "total_acc_z_train: No missing value\n",
      "\n",
      "TEST\n",
      "body_acc_x_test: No missing value\n",
      "body_acc_y_test: No missing value\n",
      "body_acc_z_test: No missing value\n",
      "body_gyro_x_test: No missing value\n",
      "body_gyro_y_test: No missing value\n",
      "body_gyro_z_test: No missing value\n",
      "total_acc_x_test: No missing value\n",
      "total_acc_y_test: No missing value\n",
      "total_acc_z_test: No missing value\n"
     ]
    }
   ],
   "source": [
    "# iterate over test and train raw files (datasets)\n",
    "for prefix in ['train', 'test']:\n",
    "\n",
    "    print('\\n'+prefix.upper())\n",
    "    parentdir = 'UCI HAR Dataset/{}/Inertial Signals/'.format(prefix)\n",
    "    # get the name of signal files in the parentdir\n",
    "    filelist = os.listdir(parentdir)\n",
    "\n",
    "    # load all the files that exist in the filelist\n",
    "    for filename in filelist:\n",
    "        # load data\n",
    "        data = datahandler.load_txt(parentdir + filename)\n",
    "        # check if there is any missing value\n",
    "        if data[data.isna().any(axis=1)].shape[0]==0:\n",
    "            print(filename[:-4]+ ': No missing value')\n",
    "        else:\n",
    "            print(filename[:-4]+ ': Missing value found')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Noise Removal\n",
    "\n",
    "Dataset description suggests that the sensor signals (accelerometer and gyroscope) are already pre-processed by applying noise filters. This information can be by the dataset description. Thus we'll skip this step."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}