FocalLoss.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "FocalLoss.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/github/aman5319/Focal-Loss/blob/master/FocalLoss.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "qXRAwoFG1zyY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Focal Loss"
      ]
    },
    {
      "metadata": {
        "id": "4D05-bc2rXyB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import torch\n",
        "from torch import nn\n",
        "import torch.nn.functional as F\n",
        "from torch.autograd import Variable"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VvHD48rg14mZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Cross Entropy Loss\n",
        "\n",
        "For Understanding Focal Loss we need to understand Cross Entropy Loss\n",
        "\n",
        "In pytorch Cross Entropy Loss = LogSoftmax + NLLLoss\n",
        "\n",
        "NLLLoss is negative log likelihood which is used in multiclass classifier.\n",
        "\n",
        "$$\n",
        "\\boldsymbol{\\mathcal{L}}=-\\frac{1}{n}\\sum_{i=1}^{n}\\log(\\hat{y}^{(i)})\n",
        "$$\n"
      ]
    },
    {
      "metadata": {
        "id": "7MGRa1lj3SgM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Below is the code to understand NLL Loss https://isaacchanghau.github.io/post/loss_functions/"
      ]
    },
    {
      "metadata": {
        "id": "AfDLlaZIbzSA",
        "colab_type": "code",
        "outputId": "dbd69448-4b72-473b-9a89-8b2659af7521",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        }
      },
      "cell_type": "code",
      "source": [
        "def NLLLoss(logs, targets):\n",
        "    out = torch.zeros_like(targets, dtype=torch.float)\n",
        "    for i in range(len(targets)):\n",
        "        out[i] = logs[i][targets[i]]\n",
        "    return -out.mean()\n",
        "\n",
        "x = torch.randn(3, 5)\n",
        "y = torch.LongTensor([4, 1, 2])\n",
        "cross_entropy_loss = torch.nn.CrossEntropyLoss()\n",
        "log_softmax = torch.nn.LogSoftmax(dim=1)\n",
        "x_log = log_softmax(x)\n",
        "\n",
        "nll_loss = torch.nn.NLLLoss()\n",
        "print(\"Torch CrossEntropyLoss: \", cross_entropy_loss(x, y))\n",
        "print(\"Torch NLL loss: \", nll_loss(x_log, y))\n",
        "print(\"Custom NLL loss: \", NLLLoss(x_log, y))\n",
        "# Torch CrossEntropyLoss:  tensor(1.8739)\n",
        "# Torch NLL loss:  tensor(1.8739)\n",
        "# Custom NLL loss:  tensor(1."
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Torch CrossEntropyLoss:  tensor(3.0131)\n",
            "Torch NLL loss:  tensor(3.0131)\n",
            "Custom NLL loss:  tensor(3.0131)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Y9yB9THmX13y",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Binary Cross Entropy\n",
        " $$\n",
        " \\boldsymbol{\\mathcal{L}} = -y*log(p) - (1-y)*log(1-p)  \n",
        " $$\n",
        " p is the probability and y is true label"
      ]
    },
    {
      "metadata": {
        "id": "AgvGBIjH3Res",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## [Focal Loss](https://arxiv.org/pdf/1708.02002.pdf)"
      ]
    },
    {
      "metadata": {
        "id": "CgoeiLRqs_Mt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Binary Focal Loss"
      ]
    },
    {
      "metadata": {
        "id": "0R9Vpp6XYii_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def binary_focal_loss(y_pred , y_true,gamma=2.0 , alpha=0.25 ,reduction=\"mean\",function=torch.sigmoid,**kwargs):\n",
        "    \"\"\"\n",
        "    Binary Version of Focal Loss\n",
        "    :args\n",
        "    \n",
        "    y_pred : prediction\n",
        "    \n",
        "    y_true : true target labels\n",
        "    \n",
        "    gamma: dampeing factor default value 2 works well according to reasearch paper\n",
        "    \n",
        "    alpha : postive to negative ratio default value 0.25 means 1 positive and 3 negative can be tuple ,list ,int and float\n",
        "    \n",
        "    reduction = mean,sum,none\n",
        "\n",
        "    function = can be sigmoid or softmax or None\n",
        "    \n",
        "    **kwargs: parameters to pass in activation function like dim in softmax\n",
        "    \n",
        "    \"\"\"\n",
        "    if isinstance(alpha,(list,tuple)):\n",
        "        pos_alpha = alpha[0] # postive sample ratio in the entire dataset\n",
        "        neg_alpha = alpha[1] #(1-alpha) # negative ratio in the entire dataset\n",
        "    elif isinstance(alpha ,(int,float)):\n",
        "        pos_alpha = alpha\n",
        "        neg_alpha = (1-alpha)\n",
        "        \n",
        "    # if else in function can be simplified be removing setting to default to sigmoid  for educational purpose\n",
        "    if function is not None:\n",
        "        y_pred = function(y_pred , **kwargs) #apply activation function\n",
        "    else :\n",
        "        assert ((y_pred <= 1) & (y_pred >= 0)).all().item() , \"negative value in y_pred value should be in the range of 0 to 1 inclusive\"\n",
        "    \n",
        "    pos_pt = torch.where(y_true==1 , y_pred , torch.ones_like(y_pred)) # positive pt (fill all the 0 place in y_true with 1 so (1-pt)=0 and log(pt)=0.0) where pt is 1\n",
        "    neg_pt = torch.where(y_true==0 , y_pred , torch.zeros_like(y_pred)) # negative pt\n",
        "    \n",
        "    pos_modulating = (1-pos_pt)**gamma # compute postive modulating factor for correct classification the value approaches to zero\n",
        "    neg_modulating = (neg_pt)**gamma # compute negative modulating factor\n",
        "    \n",
        "    \n",
        "    pos = -pos_alpha* pos_modulating*torch.log(pos_pt) #pos part\n",
        "    neg = -neg_alpha* neg_modulating*torch.log(1-neg_pt) # neg part\n",
        "    \n",
        "    loss = pos+neg  # this is final loss to be returned with some reduction\n",
        "    \n",
        "    # apply reduction\n",
        "    if reduction ==\"mean\":\n",
        "        return loss.mean()\n",
        "    elif reduction ==\"sum\":\n",
        "        return loss.sum()\n",
        "    elif reduction ==\"none\":\n",
        "        return loss # reduction mean\n",
        "    else:\n",
        "        raise f\"Wrong reduction {reduction} is choosen \\n choose one among [mean,sum,none]  \"\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "0cdd6473-cf6c-432e-a15d-33d140ed8a0f",
        "id": "n4IP6CxeignN",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "y_pred = torch.randn(32,2)\n",
        "y_true = torch.empty(32, 2).random_(2)\n",
        "F.binary_cross_entropy_with_logits(y_pred,y_true) , binary_focal_loss(y_pred,y_true,gamma=0,alpha=[1,1]) # to test the correctness of method"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(0.7474), tensor(0.7474))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "nBFl7xijeCYx",
        "colab_type": "code",
        "outputId": "9cd62753-2cd3-4d20-8fe5-7af4553b0f25",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "binary_focal_loss(y_pred,y_true) # to test the correctness of method"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(0.1416)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "metadata": {
        "id": "zmYIODDctDig",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Categorical Focal Loss\n",
        "\n",
        "This focal loss is written for only object detection is SSD and YOLO where we assume that 75% anchors are background and 25% of them are foreground\n",
        "https://blog.zenggyu.com/en/post/2018-12-05/retinanet-explained-and-demystified/#fnref10\n",
        "\n",
        "$$\n",
        "L_{cls} = -\\sum_{i=1}^{K}(y_ilog(p_i)(1-p_i)^\\gamma \\alpha_i + (1 - y_i)log(1 - p_i)p_i^\\gamma (1 - \\alpha_i))\n",
        "$$"
      ]
    },
    {
      "metadata": {
        "id": "qo8kiV3mtC-S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def sigmoid_focal_loss_cpu(logits, targets, gamma, alpha):\n",
        "    num_classes = logits.shape[1]\n",
        "    gamma = gamma\n",
        "    alpha = alpha\n",
        "    dtype = targets.dtype\n",
        "    device = targets.device\n",
        "    class_range = torch.arange(1, num_classes+1, dtype=dtype, device=device).unsqueeze(0)\n",
        "\n",
        "    t = targets.unsqueeze(1)\n",
        "    p = torch.sigmoid(logits)\n",
        "    term1 = (1 - p) ** gamma * torch.log(p)\n",
        "    term2 = p ** gamma * torch.log(1 - p)\n",
        "    return -(t == class_range).float() * term1 * alpha - ((t != class_range) * (t >= 0)).float() * term2 * (1 - alpha)\n",
        "\n",
        "\n",
        "class SigmoidFocalLoss(nn.Module):\n",
        "    def __init__(self, gamma, alpha):\n",
        "        super(SigmoidFocalLoss, self).__init__()\n",
        "        self.gamma = gamma\n",
        "        self.alpha = alpha\n",
        "\n",
        "    def forward(self, logits, targets,n_positives):\n",
        "        device = logits.device\n",
        "\n",
        "        loss = sigmoid_focal_loss_cpu(logits, targets, self.gamma, self.alpha)\n",
        "        return loss.sum()/n_positives"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hkmp1rYD94fi",
        "colab_type": "code",
        "outputId": "663d5e75-dfac-4443-f684-c72a548b1e57",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        }
      },
      "cell_type": "code",
      "source": [
        "# Generating SSD data with random configuration for testing\n",
        "\"\"\"\n",
        "confidence_prediction: (batch_size,num_anchors,num_classes) \n",
        "\"\"\"\n",
        "confidence_prediction = torch.randn((24,8732,21))\n",
        "\n",
        "\"\"\"\n",
        "target: (batch_size,num_anchors,num_classes)\n",
        "\"\"\"\n",
        "target = torch.LongTensor(24,8732,).random_(21)\n",
        "\n",
        "## Reshaping \n",
        "confidence_prediction = confidence_prediction.view(-1,21)\n",
        "print(f\"confidence_prediction shape {confidence_prediction.shape}\")\n",
        "target = target.view(-1)\n",
        "target[:208800]=0\n",
        "print(f\"target shape {target.shape}\")\n",
        "confidence_prediction.requires_grad_(True)\n",
        "confidence_prediction = confidence_prediction.cuda()\n",
        "target = target.cuda()\n",
        "(target!=0).sum()\n",
        "F.cross_entropy(confidence_prediction , target) , focal_loss(confidence_prediction , target,alpha=[1,1],gamma=0)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "confidence_prediction shape torch.Size([209568, 21])\n",
            "target shape torch.Size([209568])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(3.5079, device='cuda:0', grad_fn=<NllLossBackward>),\n",
              " tensor(3.5079, device='cuda:0', grad_fn=<DivBackward0>))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "metadata": {
        "id": "9GNBVM2Ul-7f",
        "colab_type": "code",
        "outputId": "20fe1ea7-2e04-4722-952a-df42cf2e2b31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "cell_type": "code",
      "source": [
        "# Generating SSD data with random configuration for studying the code\n",
        "\"\"\"\n",
        "confidence_prediction: (batch_size,num_anchors,num_classes) \n",
        "\"\"\"\n",
        "confidence_prediction = torch.randn((2,5,15))\n",
        "\n",
        "\"\"\"\n",
        "target: (batch_size,num_anchors,num_classes)\n",
        "\"\"\"\n",
        "target = torch.LongTensor(2,5,).random_(15)\n",
        "\n",
        "## Reshaping \n",
        "confidence_prediction = confidence_prediction.view(-1,15)\n",
        "print(f\"confidence_prediction shape {confidence_prediction.shape}\")\n",
        "target = target.view(-1)\n",
        "target[:6]=0\n",
        "print(f\"target shape {target.shape}\")\n",
        "confidence_prediction.requires_grad_(True)\n",
        "confidence_prediction = confidence_prediction.cuda()\n",
        "target = target.cuda()\n",
        "print((target!=0).sum())\n",
        "F.cross_entropy(confidence_prediction , target) , focal_loss(confidence_prediction , target)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "confidence_prediction shape torch.Size([10, 15])\n",
            "target shape torch.Size([10])\n",
            "tensor(4, device='cuda:0')\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(3.3223, device='cuda:0', grad_fn=<NllLossBackward>),\n",
              " tensor(0.8725, device='cuda:0', grad_fn=<DivBackward0>))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "metadata": {
        "id": "QsHwVdv9BL5b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Note:- When Using Focal Loss for SSD in two ways loss function can be written\n",
        "\n",
        "1. Using Sigmoid : If Sigmoid is used as a final activation layer then don't add any extra background class just use the number of classes with in the dataset. In this particular scenario binary cross entropy has be to used where 1 means foreground object and 0 means background object\n",
        "\n",
        "2.  Using Softmax:- If Softmax is used as a final activation Layer then an extra background class is to be added so for pascal voc 2007 we will 20 classes with 1 extra background class we will have total 21 class. Use Cross Entropy to calculate the loss. we are using softmax\n"
      ]
    },
    {
      "metadata": {
        "id": "WEEuf200HZBt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}