14.5 Shifting Data Through Time (Lagging and Leading).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/cover4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><em>Copyright! This material is protected, please do not copy or distribute.    by:Taher Assaf</em></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<h1 align=\"center\">Udemy course : Python Bootcamp for Data Science 2021 Numpy Pandas & Seaborn</h1> \n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5 Shifting Data Through Time (Lagging and Leading)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we import pandas library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets read a time series from a csv file using the function **pd.read_csv()**, we set the index column to be the date column using the argument **index_col**, and we convert the dates to datatime format using the argument **parse_dates = True**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple\n",
       "date              \n",
       "2019-04-01  131.24\n",
       "2019-04-02  133.25\n",
       "2019-04-03  133.35\n",
       "2019-04-04  134.16\n",
       "2019-04-05  134.49\n",
       "2019-04-06  134.09\n",
       "2019-04-07  131.72\n",
       "2019-04-08  133.27\n",
       "2019-04-09  132.88\n",
       "2019-04-10  131.24\n",
       "2019-04-11  133.25\n",
       "2019-04-12  133.35\n",
       "2019-04-13  134.16\n",
       "2019-04-14  134.49\n",
       "2019-04-15  134.09\n",
       "2019-04-16  131.72"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS = pd.read_csv('data/ex17.csv', index_col = 'date', parse_dates = True)\n",
    "TS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can shift the price **one day** backward by using the function **shift()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple\n",
       "date              \n",
       "2019-04-01     NaN\n",
       "2019-04-02  131.24\n",
       "2019-04-03  133.25\n",
       "2019-04-04  133.35\n",
       "2019-04-05  134.16\n",
       "2019-04-06  134.49\n",
       "2019-04-07  134.09\n",
       "2019-04-08  131.72\n",
       "2019-04-09  133.27\n",
       "2019-04-10  132.88\n",
       "2019-04-11  131.24\n",
       "2019-04-12  133.25\n",
       "2019-04-13  133.35\n",
       "2019-04-14  134.16\n",
       "2019-04-15  134.49\n",
       "2019-04-16  134.09"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.shift(periods = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can shift the price **two days** backward by using the function **shift()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple\n",
       "date              \n",
       "2019-04-01     NaN\n",
       "2019-04-02     NaN\n",
       "2019-04-03  131.24\n",
       "2019-04-04  133.25\n",
       "2019-04-05  133.35\n",
       "2019-04-06  134.16\n",
       "2019-04-07  134.49\n",
       "2019-04-08  134.09\n",
       "2019-04-09  131.72\n",
       "2019-04-10  133.27\n",
       "2019-04-11  132.88\n",
       "2019-04-12  131.24\n",
       "2019-04-13  133.25\n",
       "2019-04-14  133.35\n",
       "2019-04-15  134.16\n",
       "2019-04-16  134.49"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.shift(periods = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can shift the price **three days** backward by using the function **shift()**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple\n",
       "date              \n",
       "2019-04-01     NaN\n",
       "2019-04-02     NaN\n",
       "2019-04-03     NaN\n",
       "2019-04-04  131.24\n",
       "2019-04-05  133.25\n",
       "2019-04-06  133.35\n",
       "2019-04-07  134.16\n",
       "2019-04-08  134.49\n",
       "2019-04-09  134.09\n",
       "2019-04-10  131.72\n",
       "2019-04-11  133.27\n",
       "2019-04-12  132.88\n",
       "2019-04-13  131.24\n",
       "2019-04-14  133.25\n",
       "2019-04-15  133.35\n",
       "2019-04-16  134.16"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.shift(periods = 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do not need to write the name of the argument **(periods)**, we can just pass the number of periods like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple\n",
       "date              \n",
       "2019-04-01     NaN\n",
       "2019-04-02     NaN\n",
       "2019-04-03     NaN\n",
       "2019-04-04  131.24\n",
       "2019-04-05  133.25\n",
       "2019-04-06  133.35\n",
       "2019-04-07  134.16\n",
       "2019-04-08  134.49\n",
       "2019-04-09  134.09\n",
       "2019-04-10  131.72\n",
       "2019-04-11  133.27\n",
       "2019-04-12  132.88\n",
       "2019-04-13  131.24\n",
       "2019-04-14  133.25\n",
       "2019-04-15  133.35\n",
       "2019-04-16  134.16"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.shift(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make a forward shift by using **minus** for the periods. Here we use **(-1)** to shift the data one day forward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple\n",
       "date              \n",
       "2019-04-01  133.25\n",
       "2019-04-02  133.35\n",
       "2019-04-03  134.16\n",
       "2019-04-04  134.49\n",
       "2019-04-05  134.09\n",
       "2019-04-06  131.72\n",
       "2019-04-07  133.27\n",
       "2019-04-08  132.88\n",
       "2019-04-09  131.24\n",
       "2019-04-10  133.25\n",
       "2019-04-11  133.35\n",
       "2019-04-12  134.16\n",
       "2019-04-13  134.49\n",
       "2019-04-14  134.09\n",
       "2019-04-15  131.72\n",
       "2019-04-16     NaN"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.shift(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually when dealing with time series, we create a shifted data and attach it as a new column in the time series, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "      <th>lag1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-01</th>\n",
       "      <td>131.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>133.27</td>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>132.88</td>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.24</td>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple    lag1\n",
       "date                      \n",
       "2019-04-01  131.24     NaN\n",
       "2019-04-02  133.25  131.24\n",
       "2019-04-03  133.35  133.25\n",
       "2019-04-04  134.16  133.35\n",
       "2019-04-05  134.49  134.16\n",
       "2019-04-06  134.09  134.49\n",
       "2019-04-07  131.72  134.09\n",
       "2019-04-08  133.27  131.72\n",
       "2019-04-09  132.88  133.27\n",
       "2019-04-10  131.24  132.88\n",
       "2019-04-11  133.25  131.24\n",
       "2019-04-12  133.35  133.25\n",
       "2019-04-13  134.16  133.35\n",
       "2019-04-14  134.49  134.16\n",
       "2019-04-15  134.09  134.49\n",
       "2019-04-16  131.72  134.09"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS['lag1'] = TS['apple'].shift(1)\n",
    "TS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shifting data in time series generates **missing values**, we can delete these missing values using the function **dropna()**, and to make the changes reflected in the original times series we use the argument **inplace = True**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "      <th>lag1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>133.27</td>\n",
       "      <td>131.72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>132.88</td>\n",
       "      <td>133.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.24</td>\n",
       "      <td>132.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple    lag1\n",
       "date                      \n",
       "2019-04-02  133.25  131.24\n",
       "2019-04-03  133.35  133.25\n",
       "2019-04-04  134.16  133.35\n",
       "2019-04-05  134.49  134.16\n",
       "2019-04-06  134.09  134.49\n",
       "2019-04-07  131.72  134.09\n",
       "2019-04-08  133.27  131.72\n",
       "2019-04-09  132.88  133.27\n",
       "2019-04-10  131.24  132.88\n",
       "2019-04-11  133.25  131.24\n",
       "2019-04-12  133.35  133.25\n",
       "2019-04-13  134.16  133.35\n",
       "2019-04-14  134.49  134.16\n",
       "2019-04-15  134.09  134.49\n",
       "2019-04-16  131.72  134.09"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.dropna(inplace = True)\n",
    "TS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example we can calculate the daily percent change for stock prices using the **shift()** function like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "      <th>lag1</th>\n",
       "      <th>percent_change</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-02</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "      <td>0.000750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "      <td>0.006074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "      <td>0.002460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "      <td>-0.002974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "      <td>-0.017675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>133.27</td>\n",
       "      <td>131.72</td>\n",
       "      <td>0.011767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>132.88</td>\n",
       "      <td>133.27</td>\n",
       "      <td>-0.002926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.24</td>\n",
       "      <td>132.88</td>\n",
       "      <td>-0.012342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "      <td>0.015315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "      <td>0.000750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "      <td>0.006074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "      <td>0.002460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "      <td>-0.002974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "      <td>-0.017675</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple    lag1  percent_change\n",
       "date                                      \n",
       "2019-04-02  133.25  131.24             NaN\n",
       "2019-04-03  133.35  133.25        0.000750\n",
       "2019-04-04  134.16  133.35        0.006074\n",
       "2019-04-05  134.49  134.16        0.002460\n",
       "2019-04-06  134.09  134.49       -0.002974\n",
       "2019-04-07  131.72  134.09       -0.017675\n",
       "2019-04-08  133.27  131.72        0.011767\n",
       "2019-04-09  132.88  133.27       -0.002926\n",
       "2019-04-10  131.24  132.88       -0.012342\n",
       "2019-04-11  133.25  131.24        0.015315\n",
       "2019-04-12  133.35  133.25        0.000750\n",
       "2019-04-13  134.16  133.35        0.006074\n",
       "2019-04-14  134.49  134.16        0.002460\n",
       "2019-04-15  134.09  134.49       -0.002974\n",
       "2019-04-16  131.72  134.09       -0.017675"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS['percent_change'] = ( TS['apple'] / TS['apple'].shift(1) ) -1\n",
    "TS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again here we can delete the missing values like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apple</th>\n",
       "      <th>lag1</th>\n",
       "      <th>percent_change</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-04-03</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "      <td>0.000750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-04</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "      <td>0.006074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-05</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "      <td>0.002460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-06</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "      <td>-0.002974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-07</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "      <td>-0.017675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-08</th>\n",
       "      <td>133.27</td>\n",
       "      <td>131.72</td>\n",
       "      <td>0.011767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-09</th>\n",
       "      <td>132.88</td>\n",
       "      <td>133.27</td>\n",
       "      <td>-0.002926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-10</th>\n",
       "      <td>131.24</td>\n",
       "      <td>132.88</td>\n",
       "      <td>-0.012342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-11</th>\n",
       "      <td>133.25</td>\n",
       "      <td>131.24</td>\n",
       "      <td>0.015315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-12</th>\n",
       "      <td>133.35</td>\n",
       "      <td>133.25</td>\n",
       "      <td>0.000750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-13</th>\n",
       "      <td>134.16</td>\n",
       "      <td>133.35</td>\n",
       "      <td>0.006074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-14</th>\n",
       "      <td>134.49</td>\n",
       "      <td>134.16</td>\n",
       "      <td>0.002460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-15</th>\n",
       "      <td>134.09</td>\n",
       "      <td>134.49</td>\n",
       "      <td>-0.002974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-04-16</th>\n",
       "      <td>131.72</td>\n",
       "      <td>134.09</td>\n",
       "      <td>-0.017675</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             apple    lag1  percent_change\n",
       "date                                      \n",
       "2019-04-03  133.35  133.25        0.000750\n",
       "2019-04-04  134.16  133.35        0.006074\n",
       "2019-04-05  134.49  134.16        0.002460\n",
       "2019-04-06  134.09  134.49       -0.002974\n",
       "2019-04-07  131.72  134.09       -0.017675\n",
       "2019-04-08  133.27  131.72        0.011767\n",
       "2019-04-09  132.88  133.27       -0.002926\n",
       "2019-04-10  131.24  132.88       -0.012342\n",
       "2019-04-11  133.25  131.24        0.015315\n",
       "2019-04-12  133.35  133.25        0.000750\n",
       "2019-04-13  134.16  133.35        0.006074\n",
       "2019-04-14  134.49  134.16        0.002460\n",
       "2019-04-15  134.09  134.49       -0.002974\n",
       "2019-04-16  131.72  134.09       -0.017675"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TS.dropna(inplace = True)\n",
    "TS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "<h1 align=\"center\">Thank You</h1> \n",
    "\n",
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}