Ch11_PortfolioSorts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Portfolio Sorts\n",
    "\n",
    "This notebook implements univariate and bivariate portfolio sorts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Packages and Extra Functions\n",
    "\n",
    "The key functions used in this notebook are from the (local) `FinEcmt_OLS` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "MyModulePath = joinpath(pwd(),\"jlFiles\")\n",
    "!in(MyModulePath,LOAD_PATH) && push!(LOAD_PATH,MyModulePath);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "using FinEcmt_OLS, Dates, DelimitedFiles, Statistics\n",
    "using FinEcmt_TimeSeries: EMA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "The data set contains daily data for \"dates\", the equity market return, riskfree rate and the the returns of the 25 Fama-French portfolios. All returns are in percent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "The first few rows of dN, Rm and Rf\n",
      "1979-01-02     0.615     0.035\n",
      "1979-01-03     1.155     0.035\n",
      "1979-01-04     0.975     0.035\n",
      "1979-01-05     0.685     0.035\n",
      "\n",
      "size of dN, Rm, Rf, R\n",
      "(9837,)\n",
      "(9837,)\n",
      "(9837,)\n",
      "(9837, 25)\n"
     ]
    }
   ],
   "source": [
    "x   = readdlm(\"Data/MomentumSR.csv\",',')\n",
    "dN  = Date.(x[:,1],\"yyyy-mm-dd\")                  #Julia dates\n",
    "y   = convert.(Float64,x[:,2:end])\n",
    "\n",
    "(Rm,Rf,R) = (y[:,1],y[:,2],y[:,3:end])\n",
    "\n",
    "println(\"\\nThe first few rows of dN, Rm and Rf\")\n",
    "printmat([dN[1:4] Rm[1:4] Rf[1:4]])\n",
    "\n",
    "println(\"size of dN, Rm, Rf, R\")\n",
    "println(size(dN),\"\\n\",size(Rm),\"\\n\",size(Rf),\"\\n\",size(R))\n",
    "\n",
    "(T,n) = size(R);                      #number of periods and assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Univariate Sort\n",
    "\n",
    "on recent returns. \n",
    "\n",
    "The next cells create a sorting variable `X` and does a portfolio sort to create a `Lo` portfolio and a `Hi` portfolio (corresponding to the 5 lowest and highest values of `X`). The sort is done for each trading day of the sample, so the portfolios are dynamic (has time-varying portfolio weights).\n",
    "\n",
    "### A Remark on the Code\n",
    "\n",
    "- The `EMA(R,22)` function (included above) calculates a moving average of `R[t-21:t,i]` for each column `i`. We then lag thus result using `lag()`.\n",
    "- With `x = [9,7,8]`, the `rankPs(x)` function (included above) gives the output `[3,1,2]`. This says, for instance, that `7` is the lowest number (rank 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "m = 5                       #number of assets in each of Lo and Hi portfolio\n",
    "q = 22\n",
    "X = lag(EMA(R,q));      #(lag of) moving average of R\n",
    "\n",
    "#println(\"rows 2-4 of X*10:\")\n",
    "#printmat(X[2:4,:]*10,width=5,prec=2,colNames=string.(1:n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(RHi,RLo) = (fill(NaN,T),fill(NaN,T))\n",
    "for t = 2:T         #loop over periods, save portfolio returns\n",
    "    #local r, wHi, wLo          #local/global is needed in script\n",
    "    r                 = rankPs(X[t,:])  #X is lagged already, sort1[1] is index of worst asset\n",
    "    (wLo,wHi)         = (zeros(n),zeros(n))\n",
    "    wLo[r.<=m]       .= 1/m        #low rank: in Lo portfortolio\n",
    "    wHi[(n-m+1).<=r] .= 1/m        #high rank: in Hi portfolio\n",
    "    RLo[t]            = wLo'R[t,:]\n",
    "    RHi[t]            = wHi'R[t,:]\n",
    "end\n",
    "\n",
    "Rp = mean(R,dims=2);         #'passive' portfolio, equal weight on all assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Return Statistics\n",
    "\n",
    "...comparing with a passive portfolio. \n",
    "\n",
    "Excess returns for the `Lo`, `Hi` and passive (`p`) portfolios are calculated.\n",
    "\n",
    "We then use the `ReturnStats(Re,Annfactor)` function (included above) to calculate the mean (excess) return, its standard deviation and the Sharpe ratio. Annualisation is done by assuming 252 trading days per year.\n",
    "\n",
    "In calculating the return stats, we drop the first `q` observations (and thus extract observation `q+1:end`, since we use `q` observations to form the first dynamic portfolio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mStats for the portfolio returns, annualized:\u001b[22m\u001b[39m\n",
      "\n",
      "μ      3.745    14.022     9.626\n",
      "σ     19.056    17.340    16.928\n",
      "SR     0.197     0.809     0.569\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ReLo = RLo - Rf         #create excess returns\n",
    "ReHi = RHi - Rf\n",
    "Rep  = Rp - Rf          #excess return of passive portfolio\n",
    "\n",
    "Stats = ReturnStats([ReLo ReHi Rep][q+1:end,:],252)    #return stats for obs q+1 to T\n",
    "\n",
    "printblue(\"Stats for the portfolio returns, annualized:\\n\")\n",
    "printmat(Stats;colNames=[\"Lo\" \"Hi\" \"passive\"],rowNames=[\"μ\";\"σ\";\"SR\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Univariate Sort (again)\n",
    "\n",
    "on recent returns - but using another approach, which turns out to be easy to apply to the double sort as well.\n",
    "\n",
    "### A Remark on the Code\n",
    "\n",
    "- The `sortLoHi(x,v,m)` function (included above) below create n-vectors `vL` and `vH` with trues/falses indicating membership of the Lo and Hi groups.\n",
    "- As an example, `(vL,vH) = sortLoHi([3,1,2],[false,true,true],1)` gives `vL=[false,true,false]` and `vH=[false,false,true]`. This means that the 'Lo' portfolio consists of asset 2 and the 'Hi' portfolio of asset 3. Asset 1 is not assigned to any.\n",
    "\n",
    "- The `EWportf(v)` function (also from `FinEcmt_OLS`) takes such a Bool vector (eg. `vL`) and calculates (equal) portfolio weights among those assets that are `true` in `v`. It handles the case of an empty portfolio (all elements in `v` are false) by setting all portfolio weights to `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```\n",
       "sortLoHi(x,v,m)\n",
       "```\n",
       "\n",
       "Create vectors `vL` and `vH` with trues/falses indicating membership of the Lo and Hi groups. It sorts according to `x[v]`, setting the m lowest (in `vL`) and m highest values (in `vH`) to `true`. All other elements  (also those in `x[.!v]`) are set to false.\n",
       "\n",
       "### Input\n",
       "\n",
       "  * `x::Vector`:    n-vector, sorting variable\n",
       "  * `v::Vector`:    n-vector of true/false. Sorting is done within x[v]\n",
       "  * `m::Int`:       number of assets in Lo/Hi portfolio\n",
       "\n",
       "### Output\n",
       "\n",
       "  * `vL::Vector`:   n-vector of true/false, indicating membership of Lo portfolio\n",
       "  * `vH::Vector`:   n-vector of true/false, indicating membership of Hi portfolio\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@doc2 sortLoHi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#using CodeTracking\n",
    "#println(@code_string sortLoHi([1],trues(1),1))    #print the source code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mStats for the portfolio returns, annualized:\u001b[22m\u001b[39m\n",
      "\n",
      "μ      3.745    14.022     9.626\n",
      "σ     19.056    17.340    16.928\n",
      "SR     0.197     0.809     0.569\n",
      "\n",
      "\u001b[31m\u001b[1mCompare with the previous results to verify that they are the same\u001b[22m\u001b[39m\n"
     ]
    }
   ],
   "source": [
    "X = lag(EMA(R,q))                      #sort on lag of MA(q) of R\n",
    "\n",
    "m = 5                                      #m asset in each of Lo and Hi\n",
    "\n",
    "(RH,RL) = (fill(NaN,T),fill(NaN,T))\n",
    "for t = 2:T                    #loop over periods, save portfolio returns\n",
    "    #local vL,vH,wL,wH         #local/global is needed in script\n",
    "    (vL,vH) = sortLoHi(X[t,:],trues(n),m)\n",
    "    (wL,wH) = (EWportf(vL),EWportf(vH))       #portfolio weights, EW\n",
    "    RL[t]   = wL'R[t,:]\n",
    "    RH[t]   = wH'R[t,:]\n",
    "end\n",
    "\n",
    "ReL = RL - Rf\n",
    "ReH = RH - Rf       #cut out t=1, excess returns\n",
    "\n",
    "Statm = ReturnStats([ReL ReH Rep][q+1:end,:],252)\n",
    "printblue(\"Stats for the portfolio returns, annualized:\\n\")\n",
    "printmat(Stats,colNames=[\"Lo\" \"Hi\" \"passive\"],rowNames=[\"μ\";\"σ\";\"SR\"])\n",
    "\n",
    "printred(\"Compare with the previous results to verify that they are the same\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Independent Double Sort\n",
    "\n",
    "on recent volatility ($X$) and and returns ($Z$).\n",
    "\n",
    "This creates four portfolios: (Low $X$, Low $Z$), (Low $X$, High $Z$), (High $X$, Low $Z$) and (High $X$, High $Z$). Each of them is an intersection from independent sorts on $X$ and $Z.$\n",
    "\n",
    "The size of each of these portfolios can vary over time: sometimes the portfolio is empty. The portfolio weights created by the `EWportf()` function are then `NaN`.\n",
    "\n",
    "\n",
    "### A Remark on the Code\n",
    "\n",
    "- `vXL .& vZL` tests if `vXL[i]` and `vZL[i]` are both true (and then repeats for each `i`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X = lag(EMA(abs.(R),q))                #(lag of) MA of |R|, first sort variable\n",
    "Z = lag(EMA(R,q));                     #(lag of) MA of R, second sort variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(mX,mZ) = (10,10)\n",
    "\n",
    "(RLL,RLH,RHL,RHH) = [fill(NaN,T) for i=1:4]\n",
    "for t = 2:T                    #loop over periods, save portfolio returns\n",
    "    #local vXL,vXH,vZL,vZH,vLL,vLH,vHL,vHH,wLL,wLH,wHL,wHH       #local/global is needed in script\n",
    "    (vXL,vXH) = sortLoHi(X[t,:],trues(n),mX)       #in Lo/Hi according to X\n",
    "    (vZL,vZH) = sortLoHi(Z[t,:],trues(n),mZ)       #in Lo/Hi according to Z\n",
    "    vLL     = vXL .& vZL                                  #in Lo X,Low Z\n",
    "    vLH     = vXL .& vZH                                  #in Lo X, Hi Z\n",
    "    vHL     = vXH .& vZL                                  #in Hi X, Lo Z\n",
    "    vHH     = vXH .& vZH                                  #in Hi X, Hi Z\n",
    "    (wLL,wLH,wHL,wHH) = (EWportf(vLL),EWportf(vLH),EWportf(vHL),EWportf(vHH))  #portfolio weights, EW\n",
    "    (RLL[t],RLH[t],RHL[t],RHH[t])  = (wLL'R[t,:],wLH'R[t,:],wHL'R[t,:],wHH'R[t,:])\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling NaNs and Reporting Results\n",
    "\n",
    "There are NaNs in the return series, since the portfolios are sometimes empty. We need to decide on how to handle those data points. \n",
    "\n",
    "The assumption below is that an empty portfolio (when all weights are `NaNs`) means that the full investment is done in the riskfree asset and thus the excess return is zero. We thus replace the excess return of a `NaN` return with zero.\n",
    "\n",
    "We typically want to study `RLH - RLL` and `RHH - RHL` since they show the \"effect\" of the second sort variable (`Z`) while controlling for the first (`X`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mStats for the portfolio returns, annualized:\u001b[22m\u001b[39m\n",
      "\n",
      "          LL        LH        HL        HH\n",
      "μ      6.207    12.508     4.731    12.483\n",
      "σ     14.644    14.350    20.575    19.069\n",
      "SR     0.424     0.872     0.230     0.655\n",
      "\n",
      "\u001b[34m\u001b[1mStudy LH-LL and HH-HL to see the momentum effect (controlling for volatility):\u001b[22m\u001b[39m\n",
      "\n",
      "       LH-LL     HH-HL\n",
      "μ      6.301     7.752\n",
      "σ      7.878    11.195\n",
      "SR     0.800     0.692\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ReAll = [RLL RLH RHL RHH] .- Rf  #excess returns\n",
    "replace!(ReAll,NaN=>0)           #replace NaN by 0, assuming investment in riskfree\n",
    "Stats = ReturnStats(ReAll[q+1:end,:],252)\n",
    "\n",
    "printblue(\"Stats for the portfolio returns, annualized:\\n\")\n",
    "colNames = [\"LL\",\"LH\",\"HL\",\"HH\"]\n",
    "printmat(Stats,colNames=colNames,rowNames=[\"μ\";\"σ\";\"SR\"])\n",
    "\n",
    "printblue(\"Study LH-LL and HH-HL to see the momentum effect (controlling for volatility):\\n\")\n",
    "RLH_LL = ReAll[:,2] - ReAll[:,1]         #(L,H) minus (L,L)\n",
    "RHH_HL = ReAll[:,4] - ReAll[:,3]         #(H,H) minus (H,L)\n",
    "\n",
    "Stats = ReturnStats([RLH_LL RHH_HL][q+1:end,:],252)\n",
    "printmat(Stats,colNames=[\"LH-LL\",\"HH-HL\"],rowNames=[\"μ\";\"σ\";\"SR\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Dependent Double Sort\n",
    "\n",
    "on recent volatility ($X$) and and returns ($Z$).\n",
    "\n",
    "This also creates four portfolios: (Low $X$, Low $Z$), (Low $X$, High $Z$), (High $X$, Low $Z$) and (High $X$, High $Z$). However, in this case, the Low $X$ group is split up into: (Low $X$, Low $Z$) and (Low $X$, High $Z$), and similarly for the high $X$ group.\n",
    "\n",
    "The size of each of these portfolios is constant over time (unless there are missing values).\n",
    "\n",
    "### A Remark on the Code\n",
    "\n",
    "- The `vXL` from `(vXL,vXH) = sortLoHi(X[t,:],trues(n),mX)` is an n-vector where element $i$ is `true` when asset $i$ belongs to 'low according to `X[t,:]`'.\n",
    "\n",
    "- `sortLoHi(Z[t,:],vXL,mZ)` sorts `Z[t,vXL]` into low/high. The other elements in `Z[t,:]` will not belong to either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(mX,mZ) = (10,5)\n",
    "\n",
    "(RLL,RLH,RHL,RHH) = [fill(NaN,T) for i=1:4]\n",
    "for t = 2:T                    #loop over periods, save portfolio returns\n",
    "    #local vXL,vXH,vLL,vLH,vHL,vHH,wLL,wLH,wHL,wHH       #local/global is needed in script\n",
    "    (vXL,vXH) = sortLoHi(X[t,:],trues(n),mX)       #Lo/Hi according to X\n",
    "    (vLL,vLH) = sortLoHi(Z[t,:],vXL,mZ)            #within Lo X, Lo/Hi according to Z\n",
    "    (vHL,vHH) = sortLoHi(Z[t,:],vXH,mZ)            #within Hi X, Lo/Hi according to Z\n",
    "    (wLL,wLH,wHL,wHH) = (EWportf(vLL),EWportf(vLH),EWportf(vHL),EWportf(vHH))  #portfolio weights, EW\n",
    "    (RLL[t],RLH[t],RHL[t],RHH[t])  = (wLL'R[t,:],wLH'R[t,:],wHL'R[t,:],wHH'R[t,:])\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mStats for the portfolio returns, annualized:\u001b[22m\u001b[39m\n",
      "\n",
      "          LL        LH        HL        HH\n",
      "μ      7.324    12.648     5.212    13.063\n",
      "σ     15.010    14.466    20.824    19.783\n",
      "SR     0.488     0.874     0.250     0.660\n",
      "\n",
      "\u001b[34m\u001b[1mStudy LH-LL and HH-HL to see the momentum effect (controlling for volatility):\u001b[22m\u001b[39m\n",
      "\n",
      "       LH-LL     HH-HL\n",
      "μ      5.324     7.852\n",
      "σ      4.565     7.326\n",
      "SR     1.166     1.072\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ReAll = [RLL RLH RHL RHH] .- Rf\n",
    "replace!(ReAll,NaN=>0)         #replace NaN by 0, assuming investment in riskfree\n",
    "Stats = ReturnStats(ReAll[q+1:end,:],252)\n",
    "\n",
    "printblue(\"Stats for the portfolio returns, annualized:\\n\")\n",
    "colNames = [\"LL\",\"LH\",\"HL\",\"HH\"]\n",
    "printmat(Stats,colNames=colNames,rowNames=[\"μ\";\"σ\";\"SR\"])\n",
    "\n",
    "printblue(\"Study LH-LL and HH-HL to see the momentum effect (controlling for volatility):\\n\")\n",
    "RLH_LL = ReAll[:,2] - ReAll[:,1]         #(L,H) minus (L,L)\n",
    "RHH_HL = ReAll[:,4] - ReAll[:,3]         #(H,H) minus (H,L)\n",
    "\n",
    "Stats = ReturnStats([RLH_LL RHH_HL][q+1:end,:],252)\n",
    "printmat(Stats,colNames=[\"LH-LL\",\"HH-HL\"],rowNames=[\"μ\";\"σ\";\"SR\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Julia 1.10.2",
   "language": "julia",
   "name": "julia-1.10"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}