3. feature engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# import out own modules\n",
    "from modules.FeatureBuilder import *\n",
    "from modules.DataHandler import *\n",
    "from modules.SignalTransform import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why we don't simply fit intertial signals to the model?\n",
    "\n",
    "Although they showed relatively different trends between walking and stationary activity groups, we observed that most of the signals overlap within the same group activities. Hence, it's not clear whether they would be helpful for modelling activity detecting problem. Nevertheless, it's important to add that total acceleration signals showed considerable distinction even within the activities of same group. But, still, the total acceleration signals alone isn't enough to capture all the information about the event.\n",
    "\n",
    "### What we can do?\n",
    "\n",
    "Most of the signal show similar behaviour (periodic patterns) over a period of time. From the initial investigation we did on inertial signals, we know that signal values change when activity changes at some moment and it shows similar flow during this activity. Especially, we observed some fluctuations or rapid changes at the signal values for some short frames in which transaction from one activity to another activity occurs. It's important to find when these changes happen. Thus, transforming signals between time and frequency domain, we'll try to decompose the frequency of these periodic components and later find possible significant peaks in frequency spectra. For this we'll use these concepts:\n",
    "    \n",
    "   1. **Fast Fourier Transform (FFT)** - Used for finding frequency of periodical components\n",
    "   2. **Power Spectral Density (PSD)** - Finds peaks corrensponds to power distribution at that frequency\n",
    "   3. **Atocorrelation (aCORR)** - Caculates the serial correlation of a signal with its lagged signal\n",
    "   \n",
    "   \n",
    "   For more information on implementation of these mehotds, please check out *'modules.SignalTransform'*.\n",
    "\n",
    "### What is next now?\n",
    "\n",
    "Now we have extra signals to conduct more detailed feature engineering which may convey better information. We'll use these transformation signals addition to initial inertial signals for doing feature extraction. Our total features set we'll be consist of features calculated on this newly created signal and those calculated on normal inertial signals.\n",
    "\n",
    "\n",
    "### Main features\n",
    "\n",
    "main_features: those are the features that we'll be calculated on original inertial signals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "featureBuilder = FeatureBuilder(n_peaks=2)\n",
    "featureBuilder.init_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['std', 'mean', 'mad', 'max', 'min', 'iqr', 'correlation-1', 'correlation-2']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "featureBuilder.get_main_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Descrioption of main features\n",
    "\n",
    "* 'mean' mean vale\n",
    "* 'std': standart deviation\n",
    "* 'mad': median absolute deviation\n",
    "* 'max': larget value in array\n",
    "* 'min': smallest value in array\n",
    "* 'iqr': interquartile range\n",
    "* 'correlation-1': correlation\n",
    "* 'correlation-2': correlation\n",
    "\n",
    "We know that we have 3 different signals and each signal is represented on 3-axis. So if the given signal is on the x-axis, then *'correlation-1'* and *'correlation-2'* show correlation between x-y and x-z. It the same for the other axes.\n",
    "\n",
    "\n",
    "### Domain  features\n",
    "\n",
    "domain_features: those are the features that we'll be calculated from signal transformations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aCORR-peaks-mean',\n",
       " 'aCORR-peak-value-0',\n",
       " 'aCORR-peak-value-1',\n",
       " 'aCORR-peak-domain-0',\n",
       " 'aCORR-peak-domain-1',\n",
       " 'PSD-peaks-mean',\n",
       " 'PSD-peak-value-0',\n",
       " 'PSD-peak-value-1',\n",
       " 'PSD-peak-domain-0',\n",
       " 'PSD-peak-domain-1',\n",
       " 'FFT-peaks-mean',\n",
       " 'FFT-peak-value-0',\n",
       " 'FFT-peak-value-1',\n",
       " 'FFT-peak-domain-0',\n",
       " 'FFT-peak-domain-1']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "featureBuilder.get_domain_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Description of domain featues\n",
    "\n",
    "for each of FFT, PSD, and aCORR we have the same structue below:\n",
    "\n",
    "* 'peaks-mean': mean of the first n selected peaks-value (not domains)\n",
    "* 'peak-value':{} : e.g. for the first 2 selected peaks, it will be like {'1':0, '2':0} \n",
    "* 'peak-domain': {}: e.g for the first 2 selected peaks, it will be like {'1':0, '2':0}\n",
    "\n",
    "\n",
    "For each transformation, we'll look at the first n peaks in the signal. We're not only the interested in the amplitude of these peaks happened, but also where/when this peaks happened in the t/f-domains. Because moving where this peak occurs can also be helpful for discrimination period pattern. Thus we'll not only take first n peaks of transformations, but also consider their t/f-domains.\n",
    "\n",
    "So *'peak-value'* stores the amplitude of the first n peaks, *'peak-domain'* stores the information about at what frequency/time domain this peaks happens. In the above example n_peak=2, therefore, two peak-related features has generated.\n",
    "\n",
    "## Feature Generation\n",
    "\n",
    "Now, here comes the fun part, finally! In this part, will carry out feature engineering on test/train internal signals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Init our modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init datahandler module\n",
    "dataHandler = DataHandler()\n",
    "\n",
    "# init SignalTransform\n",
    "# N: reading size for given window\n",
    "# F: sampling rate with Hz\n",
    "# t: size of fixed-width sliding wondow in seconds\n",
    "signalTransform = SignalTransform(N=128, F=50, t=2.56)\n",
    "\n",
    "# init FeatureBuilder\n",
    "featureBuilder = FeatureBuilder(n_peaks=5)\n",
    "featureBuilder.init_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create feature labels\n",
    "Create features labels from the list of features that are initialized in FutureBuilder model. We will use them as dataset labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['body_acc_x_train.txt',\n",
       " 'body_acc_y_train.txt',\n",
       " 'body_acc_z_train.txt',\n",
       " 'body_gyro_x_train.txt',\n",
       " 'body_gyro_y_train.txt',\n",
       " 'body_gyro_z_train.txt',\n",
       " 'total_acc_x_train.txt',\n",
       " 'total_acc_y_train.txt',\n",
       " 'total_acc_z_train.txt']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the name of signal files in the path\n",
    "signal_names = os.listdir('UCI HAR Dataset/train/Inertial Signals/')\n",
    "\n",
    "signal_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BODY_ACC_X_STD',\n",
       " 'BODY_ACC_X_MEAN',\n",
       " 'BODY_ACC_X_MAD',\n",
       " 'BODY_ACC_X_MAX',\n",
       " 'BODY_ACC_X_MIN',\n",
       " 'BODY_ACC_X_IQR',\n",
       " 'BODY_ACC_X_CORRELATION-1',\n",
       " 'BODY_ACC_X_CORRELATION-2',\n",
       " 'BODY_ACC_X_ACORR-PEAKS-MEAN',\n",
       " 'BODY_ACC_X_ACORR-PEAK-VALUE-0']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# list stores feature labels\n",
    "feature_labels = []\n",
    "\n",
    "for signal in signal_names:\n",
    "\n",
    "    main_features = [(signal[:-9]+name).upper() for name in featureBuilder.get_main_features()]\n",
    "    domain_features = [(signal[:-9]+name).upper() for name in featureBuilder.get_domain_features()]\n",
    "    \n",
    "    feature_labels.extend(main_features)\n",
    "    feature_labels.extend(domain_features)\n",
    "    \n",
    "# also add target label\n",
    "feature_labels.append('ACTIVITY')\n",
    "    \n",
    "feature_labels[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# given signal calculate both main and domain features and return values\n",
    "def get_all_features(signal, corr_signals):\n",
    "    '''\n",
    "    input:\n",
    "        signal: signal array for a axis in which caculation is will be done\n",
    "        corr_signals: list that contains signal arrays for other two axes,\n",
    "                      these will be used for calculation correlations.\n",
    "                      e.g. if signal is x, then corr_signals=[z,y]\n",
    "                      \n",
    "    output: return 1D array which contains both main and domain features \n",
    "    '''\n",
    "\n",
    "    # init/reste fature values before start\n",
    "    featureBuilder.init_features()\n",
    "    \n",
    "    # calculate main features\n",
    "    featureBuilder.calculate_main_features(x_signal, corr_signals)\n",
    "\n",
    "    # do time-frequency domain transformations\n",
    "    domain_fft, signal_fft = signalTransform.fft_transform(x_signal)\n",
    "    domain_psd, signal_psd = signalTransform.psd_transform(x_signal)\n",
    "    domain_aCorr, signal_aCorr = signalTransform.aCorr_transform(x_signal)\n",
    "\n",
    "    # calculate domain features on different transformations\n",
    "    featureBuilder.calculate_domain_features(domain_fft, signal_fft, t_name='FFT')\n",
    "    featureBuilder.calculate_domain_features(domain_psd, signal_psd, t_name='PSD')\n",
    "    featureBuilder.calculate_domain_features(domain_aCorr, signal_aCorr, t_name='aCORR')\n",
    "\n",
    "    # get features and concotanate them\n",
    "    main_features = featureBuilder.get_main_features(return_values=True)\n",
    "    domain_features = featureBuilder.get_domain_features(return_values=True)\n",
    "    \n",
    "    return np.concatenate((main_features, domain_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# accumulator for sotring data\n",
    "\n",
    "for prefix in ['train', 'test']:\n",
    "    \n",
    "    data = []\n",
    "    \n",
    "    # load initial data\n",
    "    X_data = dataHandler.load_files('UCI HAR Dataset/{p}/Inertial Signals/'.format(p=prefix))\n",
    "    y_data = dataHandler.load_txt('UCI HAR Dataset/{p}/y_{p}.txt'.format(p=prefix)).values\n",
    "\n",
    "    for row in range(X_data.shape[0]):\n",
    "\n",
    "        # accumulator for storing features at each row\n",
    "        features = []\n",
    "\n",
    "        # iterate over signal types by +3, becase for each signal type we have 3 differnet axes values\n",
    "        for signal in range(0, X_data.shape[2], 3):\n",
    "\n",
    "            # fet the signal for each axis\n",
    "            x_signal = X_data[row][:, signal]\n",
    "            y_signal = X_data[row][:, signal+1]\n",
    "            z_signal = X_data[row][:, signal+2]\n",
    "\n",
    "            # GIVEN SIGNAL CALCULATE BOTH MAIN AND DOMAIN FEATURES \n",
    "            # corr_siganls is used to calculate corrlation of theese singals with the given signal\n",
    "            x_features = get_all_features(x_signal, corr_signals=[y_signal, z_signal])\n",
    "            y_features = get_all_features(y_signal, corr_signals=[x_signal, z_signal])\n",
    "            z_features = get_all_features(z_signal, corr_signals=[x_signal, y_signal])\n",
    "\n",
    "            # complete features for each row\n",
    "            features.append(np.concatenate((x_features, y_features, z_features)))\n",
    "\n",
    "        # add new feture row to the data list \n",
    "        data.append(np.array(features).flatten())\n",
    "\n",
    "    # save data, add y_data to data and ACTIVITY to labes as well\n",
    "    dataframe = pd.DataFrame(np.hstack((data, y_data)), columns=feature_labels)\n",
    "    dataframe.to_csv('dataset/{}.csv'.format(prefix), index=False, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Engineered data\n",
    "\n",
    "Feature engineering part is now done. Our data looks like below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BODY_ACC_X_STD</th>\n",
       "      <th>BODY_ACC_X_MEAN</th>\n",
       "      <th>BODY_ACC_X_MAD</th>\n",
       "      <th>BODY_ACC_X_MAX</th>\n",
       "      <th>BODY_ACC_X_MIN</th>\n",
       "      <th>BODY_ACC_X_IQR</th>\n",
       "      <th>BODY_ACC_X_CORRELATION-1</th>\n",
       "      <th>BODY_ACC_X_CORRELATION-2</th>\n",
       "      <th>BODY_ACC_X_ACORR-PEAKS-MEAN</th>\n",
       "      <th>BODY_ACC_X_ACORR-PEAK-VALUE-0</th>\n",
       "      <th>...</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-VALUE-1</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-VALUE-2</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-VALUE-3</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-VALUE-4</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-DOMAIN-0</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-DOMAIN-1</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-DOMAIN-2</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-DOMAIN-3</th>\n",
       "      <th>TOTAL_ACC_Z_FFT-PEAK-DOMAIN-4</th>\n",
       "      <th>ACTIVITY</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.002941</td>\n",
       "      <td>0.002269</td>\n",
       "      <td>0.002025</td>\n",
       "      <td>0.010810</td>\n",
       "      <td>-0.004294</td>\n",
       "      <td>0.004812</td>\n",
       "      <td>0.374934</td>\n",
       "      <td>0.433372</td>\n",
       "      <td>0.001125</td>\n",
       "      <td>0.001205</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000781</td>\n",
       "      <td>0.000479</td>\n",
       "      <td>0.000556</td>\n",
       "      <td>0.000576</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>3.968254</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>5.952381</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.001981</td>\n",
       "      <td>0.000174</td>\n",
       "      <td>0.000110</td>\n",
       "      <td>0.005251</td>\n",
       "      <td>-0.006706</td>\n",
       "      <td>0.001970</td>\n",
       "      <td>-0.011562</td>\n",
       "      <td>-0.071672</td>\n",
       "      <td>0.000038</td>\n",
       "      <td>0.000031</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000660</td>\n",
       "      <td>0.000661</td>\n",
       "      <td>0.000582</td>\n",
       "      <td>0.000858</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.968254</td>\n",
       "      <td>5.158730</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.002908</td>\n",
       "      <td>0.000428</td>\n",
       "      <td>0.000627</td>\n",
       "      <td>0.008167</td>\n",
       "      <td>-0.010483</td>\n",
       "      <td>0.003138</td>\n",
       "      <td>-0.121905</td>\n",
       "      <td>-0.179492</td>\n",
       "      <td>0.000097</td>\n",
       "      <td>0.000255</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000832</td>\n",
       "      <td>0.000626</td>\n",
       "      <td>0.000790</td>\n",
       "      <td>0.001252</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.587302</td>\n",
       "      <td>2.380952</td>\n",
       "      <td>4.365079</td>\n",
       "      <td>5.952381</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.002678</td>\n",
       "      <td>0.000329</td>\n",
       "      <td>0.000269</td>\n",
       "      <td>0.008167</td>\n",
       "      <td>-0.010483</td>\n",
       "      <td>0.003128</td>\n",
       "      <td>-0.301393</td>\n",
       "      <td>-0.360048</td>\n",
       "      <td>0.000181</td>\n",
       "      <td>0.000304</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000749</td>\n",
       "      <td>0.000358</td>\n",
       "      <td>0.000727</td>\n",
       "      <td>0.000890</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>4.365079</td>\n",
       "      <td>5.952381</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.002015</td>\n",
       "      <td>-0.000195</td>\n",
       "      <td>-0.000144</td>\n",
       "      <td>0.005650</td>\n",
       "      <td>-0.006847</td>\n",
       "      <td>0.002622</td>\n",
       "      <td>-0.152752</td>\n",
       "      <td>-0.188102</td>\n",
       "      <td>0.000052</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000890</td>\n",
       "      <td>0.000660</td>\n",
       "      <td>0.000693</td>\n",
       "      <td>0.000565</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.968254</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>5.952381</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.002276</td>\n",
       "      <td>-0.000078</td>\n",
       "      <td>-0.000182</td>\n",
       "      <td>0.005650</td>\n",
       "      <td>-0.006847</td>\n",
       "      <td>0.002883</td>\n",
       "      <td>-0.206476</td>\n",
       "      <td>-0.149615</td>\n",
       "      <td>0.000045</td>\n",
       "      <td>-0.000060</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001155</td>\n",
       "      <td>0.001041</td>\n",
       "      <td>0.000690</td>\n",
       "      <td>0.000885</td>\n",
       "      <td>1.587302</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>5.555556</td>\n",
       "      <td>7.142857</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.002409</td>\n",
       "      <td>0.000387</td>\n",
       "      <td>0.000461</td>\n",
       "      <td>0.006637</td>\n",
       "      <td>-0.005558</td>\n",
       "      <td>0.003317</td>\n",
       "      <td>-0.093299</td>\n",
       "      <td>-0.134364</td>\n",
       "      <td>0.000103</td>\n",
       "      <td>0.000155</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001078</td>\n",
       "      <td>0.001735</td>\n",
       "      <td>0.000898</td>\n",
       "      <td>0.000974</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>6.746032</td>\n",
       "      <td>9.126984</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.002527</td>\n",
       "      <td>-0.000030</td>\n",
       "      <td>-0.000261</td>\n",
       "      <td>0.006637</td>\n",
       "      <td>-0.006030</td>\n",
       "      <td>0.003928</td>\n",
       "      <td>-0.158534</td>\n",
       "      <td>-0.016404</td>\n",
       "      <td>0.000093</td>\n",
       "      <td>0.000120</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000659</td>\n",
       "      <td>0.001169</td>\n",
       "      <td>0.000670</td>\n",
       "      <td>0.000786</td>\n",
       "      <td>1.587302</td>\n",
       "      <td>2.380952</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>5.158730</td>\n",
       "      <td>6.349206</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.002278</td>\n",
       "      <td>-0.000058</td>\n",
       "      <td>-0.000261</td>\n",
       "      <td>0.006897</td>\n",
       "      <td>-0.006030</td>\n",
       "      <td>0.002976</td>\n",
       "      <td>-0.161664</td>\n",
       "      <td>-0.032960</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>0.000138</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000448</td>\n",
       "      <td>0.001132</td>\n",
       "      <td>0.000402</td>\n",
       "      <td>0.000287</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>5.158730</td>\n",
       "      <td>5.952381</td>\n",
       "      <td>6.746032</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.003095</td>\n",
       "      <td>0.000620</td>\n",
       "      <td>0.000495</td>\n",
       "      <td>0.007276</td>\n",
       "      <td>-0.009268</td>\n",
       "      <td>0.003705</td>\n",
       "      <td>-0.173019</td>\n",
       "      <td>-0.444580</td>\n",
       "      <td>0.000278</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001386</td>\n",
       "      <td>0.001759</td>\n",
       "      <td>0.000503</td>\n",
       "      <td>0.000710</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>6.349206</td>\n",
       "      <td>7.142857</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.003091</td>\n",
       "      <td>-0.000143</td>\n",
       "      <td>-0.000448</td>\n",
       "      <td>0.007276</td>\n",
       "      <td>-0.009268</td>\n",
       "      <td>0.003881</td>\n",
       "      <td>0.010187</td>\n",
       "      <td>-0.353347</td>\n",
       "      <td>0.000220</td>\n",
       "      <td>0.000251</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001877</td>\n",
       "      <td>0.001726</td>\n",
       "      <td>0.000635</td>\n",
       "      <td>0.000989</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>5.952381</td>\n",
       "      <td>6.746032</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.001981</td>\n",
       "      <td>-0.000278</td>\n",
       "      <td>-0.000063</td>\n",
       "      <td>0.004692</td>\n",
       "      <td>-0.005895</td>\n",
       "      <td>0.002967</td>\n",
       "      <td>-0.263259</td>\n",
       "      <td>-0.117698</td>\n",
       "      <td>0.000039</td>\n",
       "      <td>0.000206</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000704</td>\n",
       "      <td>0.000435</td>\n",
       "      <td>0.000675</td>\n",
       "      <td>0.000578</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>5.555556</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.001693</td>\n",
       "      <td>0.000182</td>\n",
       "      <td>0.000594</td>\n",
       "      <td>0.004692</td>\n",
       "      <td>-0.004877</td>\n",
       "      <td>0.002367</td>\n",
       "      <td>-0.033188</td>\n",
       "      <td>-0.154827</td>\n",
       "      <td>0.000034</td>\n",
       "      <td>0.000151</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000574</td>\n",
       "      <td>0.000358</td>\n",
       "      <td>0.000267</td>\n",
       "      <td>0.000250</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.587302</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>4.365079</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.001676</td>\n",
       "      <td>-0.000083</td>\n",
       "      <td>0.000167</td>\n",
       "      <td>0.005081</td>\n",
       "      <td>-0.004899</td>\n",
       "      <td>0.002563</td>\n",
       "      <td>0.045010</td>\n",
       "      <td>-0.142443</td>\n",
       "      <td>0.000049</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000551</td>\n",
       "      <td>0.000494</td>\n",
       "      <td>0.000527</td>\n",
       "      <td>0.000612</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>4.365079</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.005089</td>\n",
       "      <td>0.004198</td>\n",
       "      <td>0.002676</td>\n",
       "      <td>0.016638</td>\n",
       "      <td>-0.007131</td>\n",
       "      <td>0.007588</td>\n",
       "      <td>0.816781</td>\n",
       "      <td>0.734245</td>\n",
       "      <td>0.002785</td>\n",
       "      <td>0.004017</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001057</td>\n",
       "      <td>0.000798</td>\n",
       "      <td>0.001521</td>\n",
       "      <td>0.001014</td>\n",
       "      <td>1.190476</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>4.365079</td>\n",
       "      <td>5.555556</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.002436</td>\n",
       "      <td>0.000335</td>\n",
       "      <td>0.000277</td>\n",
       "      <td>0.006151</td>\n",
       "      <td>-0.007131</td>\n",
       "      <td>0.003353</td>\n",
       "      <td>0.328602</td>\n",
       "      <td>0.237321</td>\n",
       "      <td>0.000044</td>\n",
       "      <td>0.000068</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001076</td>\n",
       "      <td>0.000624</td>\n",
       "      <td>0.000916</td>\n",
       "      <td>0.000598</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>3.968254</td>\n",
       "      <td>5.158730</td>\n",
       "      <td>6.746032</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.002403</td>\n",
       "      <td>0.000301</td>\n",
       "      <td>0.000293</td>\n",
       "      <td>0.006151</td>\n",
       "      <td>-0.004542</td>\n",
       "      <td>0.003420</td>\n",
       "      <td>0.279916</td>\n",
       "      <td>0.127040</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>0.000089</td>\n",
       "      <td>...</td>\n",
       "      <td>0.001077</td>\n",
       "      <td>0.000924</td>\n",
       "      <td>0.000427</td>\n",
       "      <td>0.000870</td>\n",
       "      <td>2.380952</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>5.158730</td>\n",
       "      <td>6.746032</td>\n",
       "      <td>7.539683</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.002159</td>\n",
       "      <td>0.000527</td>\n",
       "      <td>0.000547</td>\n",
       "      <td>0.005911</td>\n",
       "      <td>-0.006346</td>\n",
       "      <td>0.002698</td>\n",
       "      <td>0.153840</td>\n",
       "      <td>-0.096417</td>\n",
       "      <td>0.000060</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000678</td>\n",
       "      <td>0.000960</td>\n",
       "      <td>0.001006</td>\n",
       "      <td>0.000848</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.587302</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>3.571429</td>\n",
       "      <td>4.761905</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.002225</td>\n",
       "      <td>0.000032</td>\n",
       "      <td>0.000276</td>\n",
       "      <td>0.004050</td>\n",
       "      <td>-0.008133</td>\n",
       "      <td>0.002943</td>\n",
       "      <td>0.191039</td>\n",
       "      <td>-0.044138</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>-0.000023</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000820</td>\n",
       "      <td>0.000792</td>\n",
       "      <td>0.001411</td>\n",
       "      <td>0.000879</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.587302</td>\n",
       "      <td>2.380952</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>4.365079</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.002121</td>\n",
       "      <td>-0.000414</td>\n",
       "      <td>-0.000118</td>\n",
       "      <td>0.003522</td>\n",
       "      <td>-0.008133</td>\n",
       "      <td>0.002399</td>\n",
       "      <td>0.111770</td>\n",
       "      <td>0.074668</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>0.000057</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000749</td>\n",
       "      <td>0.000916</td>\n",
       "      <td>0.000931</td>\n",
       "      <td>0.000631</td>\n",
       "      <td>0.793651</td>\n",
       "      <td>1.984127</td>\n",
       "      <td>3.174603</td>\n",
       "      <td>3.968254</td>\n",
       "      <td>5.555556</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20 rows × 370 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    BODY_ACC_X_STD  BODY_ACC_X_MEAN  BODY_ACC_X_MAD  BODY_ACC_X_MAX  \\\n",
       "0         0.002941         0.002269        0.002025        0.010810   \n",
       "1         0.001981         0.000174        0.000110        0.005251   \n",
       "2         0.002908         0.000428        0.000627        0.008167   \n",
       "3         0.002678         0.000329        0.000269        0.008167   \n",
       "4         0.002015        -0.000195       -0.000144        0.005650   \n",
       "5         0.002276        -0.000078       -0.000182        0.005650   \n",
       "6         0.002409         0.000387        0.000461        0.006637   \n",
       "7         0.002527        -0.000030       -0.000261        0.006637   \n",
       "8         0.002278        -0.000058       -0.000261        0.006897   \n",
       "9         0.003095         0.000620        0.000495        0.007276   \n",
       "10        0.003091        -0.000143       -0.000448        0.007276   \n",
       "11        0.001981        -0.000278       -0.000063        0.004692   \n",
       "12        0.001693         0.000182        0.000594        0.004692   \n",
       "13        0.001676        -0.000083        0.000167        0.005081   \n",
       "14        0.005089         0.004198        0.002676        0.016638   \n",
       "15        0.002436         0.000335        0.000277        0.006151   \n",
       "16        0.002403         0.000301        0.000293        0.006151   \n",
       "17        0.002159         0.000527        0.000547        0.005911   \n",
       "18        0.002225         0.000032        0.000276        0.004050   \n",
       "19        0.002121        -0.000414       -0.000118        0.003522   \n",
       "\n",
       "    BODY_ACC_X_MIN  BODY_ACC_X_IQR  BODY_ACC_X_CORRELATION-1  \\\n",
       "0        -0.004294        0.004812                  0.374934   \n",
       "1        -0.006706        0.001970                 -0.011562   \n",
       "2        -0.010483        0.003138                 -0.121905   \n",
       "3        -0.010483        0.003128                 -0.301393   \n",
       "4        -0.006847        0.002622                 -0.152752   \n",
       "5        -0.006847        0.002883                 -0.206476   \n",
       "6        -0.005558        0.003317                 -0.093299   \n",
       "7        -0.006030        0.003928                 -0.158534   \n",
       "8        -0.006030        0.002976                 -0.161664   \n",
       "9        -0.009268        0.003705                 -0.173019   \n",
       "10       -0.009268        0.003881                  0.010187   \n",
       "11       -0.005895        0.002967                 -0.263259   \n",
       "12       -0.004877        0.002367                 -0.033188   \n",
       "13       -0.004899        0.002563                  0.045010   \n",
       "14       -0.007131        0.007588                  0.816781   \n",
       "15       -0.007131        0.003353                  0.328602   \n",
       "16       -0.004542        0.003420                  0.279916   \n",
       "17       -0.006346        0.002698                  0.153840   \n",
       "18       -0.008133        0.002943                  0.191039   \n",
       "19       -0.008133        0.002399                  0.111770   \n",
       "\n",
       "    BODY_ACC_X_CORRELATION-2  BODY_ACC_X_ACORR-PEAKS-MEAN  \\\n",
       "0                   0.433372                     0.001125   \n",
       "1                  -0.071672                     0.000038   \n",
       "2                  -0.179492                     0.000097   \n",
       "3                  -0.360048                     0.000181   \n",
       "4                  -0.188102                     0.000052   \n",
       "5                  -0.149615                     0.000045   \n",
       "6                  -0.134364                     0.000103   \n",
       "7                  -0.016404                     0.000093   \n",
       "8                  -0.032960                     0.000111   \n",
       "9                  -0.444580                     0.000278   \n",
       "10                 -0.353347                     0.000220   \n",
       "11                 -0.117698                     0.000039   \n",
       "12                 -0.154827                     0.000034   \n",
       "13                 -0.142443                     0.000049   \n",
       "14                  0.734245                     0.002785   \n",
       "15                  0.237321                     0.000044   \n",
       "16                  0.127040                     0.000037   \n",
       "17                 -0.096417                     0.000060   \n",
       "18                 -0.044138                     0.000019   \n",
       "19                  0.074668                     0.000022   \n",
       "\n",
       "    BODY_ACC_X_ACORR-PEAK-VALUE-0  ...  TOTAL_ACC_Z_FFT-PEAK-VALUE-1  \\\n",
       "0                        0.001205  ...                      0.000781   \n",
       "1                        0.000031  ...                      0.000660   \n",
       "2                        0.000255  ...                      0.000832   \n",
       "3                        0.000304  ...                      0.000749   \n",
       "4                        0.000060  ...                      0.000890   \n",
       "5                       -0.000060  ...                      0.001155   \n",
       "6                        0.000155  ...                      0.001078   \n",
       "7                        0.000120  ...                      0.000659   \n",
       "8                        0.000138  ...                      0.000448   \n",
       "9                        0.000231  ...                      0.001386   \n",
       "10                       0.000251  ...                      0.001877   \n",
       "11                       0.000206  ...                      0.000704   \n",
       "12                       0.000151  ...                      0.000574   \n",
       "13                       0.000015  ...                      0.000551   \n",
       "14                       0.004017  ...                      0.001057   \n",
       "15                       0.000068  ...                      0.001076   \n",
       "16                       0.000089  ...                      0.001077   \n",
       "17                       0.000041  ...                      0.000678   \n",
       "18                      -0.000023  ...                      0.000820   \n",
       "19                       0.000057  ...                      0.000749   \n",
       "\n",
       "    TOTAL_ACC_Z_FFT-PEAK-VALUE-2  TOTAL_ACC_Z_FFT-PEAK-VALUE-3  \\\n",
       "0                       0.000479                      0.000556   \n",
       "1                       0.000661                      0.000582   \n",
       "2                       0.000626                      0.000790   \n",
       "3                       0.000358                      0.000727   \n",
       "4                       0.000660                      0.000693   \n",
       "5                       0.001041                      0.000690   \n",
       "6                       0.001735                      0.000898   \n",
       "7                       0.001169                      0.000670   \n",
       "8                       0.001132                      0.000402   \n",
       "9                       0.001759                      0.000503   \n",
       "10                      0.001726                      0.000635   \n",
       "11                      0.000435                      0.000675   \n",
       "12                      0.000358                      0.000267   \n",
       "13                      0.000494                      0.000527   \n",
       "14                      0.000798                      0.001521   \n",
       "15                      0.000624                      0.000916   \n",
       "16                      0.000924                      0.000427   \n",
       "17                      0.000960                      0.001006   \n",
       "18                      0.000792                      0.001411   \n",
       "19                      0.000916                      0.000931   \n",
       "\n",
       "    TOTAL_ACC_Z_FFT-PEAK-VALUE-4  TOTAL_ACC_Z_FFT-PEAK-DOMAIN-0  \\\n",
       "0                       0.000576                       1.190476   \n",
       "1                       0.000858                       0.793651   \n",
       "2                       0.001252                       0.793651   \n",
       "3                       0.000890                       1.190476   \n",
       "4                       0.000565                       1.190476   \n",
       "5                       0.000885                       1.587302   \n",
       "6                       0.000974                       1.190476   \n",
       "7                       0.000786                       1.587302   \n",
       "8                       0.000287                       1.190476   \n",
       "9                       0.000710                       1.984127   \n",
       "10                      0.000989                       1.190476   \n",
       "11                      0.000578                       0.793651   \n",
       "12                      0.000250                       0.793651   \n",
       "13                      0.000612                       0.793651   \n",
       "14                      0.001014                       1.190476   \n",
       "15                      0.000598                       1.984127   \n",
       "16                      0.000870                       2.380952   \n",
       "17                      0.000848                       0.793651   \n",
       "18                      0.000879                       0.793651   \n",
       "19                      0.000631                       0.793651   \n",
       "\n",
       "    TOTAL_ACC_Z_FFT-PEAK-DOMAIN-1  TOTAL_ACC_Z_FFT-PEAK-DOMAIN-2  \\\n",
       "0                        3.174603                       3.968254   \n",
       "1                        1.984127                       2.777778   \n",
       "2                        1.587302                       2.380952   \n",
       "3                        1.984127                       3.571429   \n",
       "4                        2.777778                       3.968254   \n",
       "5                        2.777778                       3.571429   \n",
       "6                        2.777778                       3.571429   \n",
       "7                        2.380952                       3.571429   \n",
       "8                        2.777778                       5.158730   \n",
       "9                        3.174603                       4.761905   \n",
       "10                       3.174603                       4.761905   \n",
       "11                       1.984127                       3.571429   \n",
       "12                       1.587302                       2.777778   \n",
       "13                       1.984127                       2.777778   \n",
       "14                       1.984127                       2.777778   \n",
       "15                       3.174603                       3.968254   \n",
       "16                       3.174603                       5.158730   \n",
       "17                       1.587302                       2.777778   \n",
       "18                       1.587302                       2.380952   \n",
       "19                       1.984127                       3.174603   \n",
       "\n",
       "    TOTAL_ACC_Z_FFT-PEAK-DOMAIN-3  TOTAL_ACC_Z_FFT-PEAK-DOMAIN-4  ACTIVITY  \n",
       "0                        4.761905                       5.952381       5.0  \n",
       "1                        3.968254                       5.158730       5.0  \n",
       "2                        4.365079                       5.952381       5.0  \n",
       "3                        4.365079                       5.952381       5.0  \n",
       "4                        4.761905                       5.952381       5.0  \n",
       "5                        5.555556                       7.142857       5.0  \n",
       "6                        6.746032                       9.126984       5.0  \n",
       "7                        5.158730                       6.349206       5.0  \n",
       "8                        5.952381                       6.746032       5.0  \n",
       "9                        6.349206                       7.142857       5.0  \n",
       "10                       5.952381                       6.746032       5.0  \n",
       "11                       4.761905                       5.555556       5.0  \n",
       "12                       3.571429                       4.365079       5.0  \n",
       "13                       3.571429                       4.365079       5.0  \n",
       "14                       4.365079                       5.555556       5.0  \n",
       "15                       5.158730                       6.746032       5.0  \n",
       "16                       6.746032                       7.539683       5.0  \n",
       "17                       3.571429                       4.761905       5.0  \n",
       "18                       3.174603                       4.365079       5.0  \n",
       "19                       3.968254                       5.555556       5.0  \n",
       "\n",
       "[20 rows x 370 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('dataset/train.csv').head(20)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}