M_1B_activation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "435ffd9d",
   "metadata": {},
   "source": [
    "# Activation Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cf12bbd7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-10  -9  -8  -7  -6  -5  -4  -3  -2  -1   0   1   2   3   4   5   6   7\n",
      "   8   9  10]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "  \n",
    "x = np.arange(-10, 11, 1)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4d83b236",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_graph(y,ylabel):\n",
    "    plt.figure()\n",
    "    plt.plot(x,y, 'o--')\n",
    "    plt.grid(True)\n",
    "    plt.xlabel(\"X\")\n",
    "    plt.ylabel(ylabel)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c7be4b",
   "metadata": {},
   "source": [
    "## Classic Activation Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da6f5245",
   "metadata": {},
   "source": [
    "### Step function\n",
    "\n",
    "The step or threshold activation function is a simple activation function. In the year 1943 McCulloch & Pitts [[@mcculloch1943logical]](https://link.springer.com/article/10.1007/BF02478259) introduced the original perceptron and used a step activation function like the following equation\n",
    " \n",
    "$$\n",
    "f(x) =\n",
    "  \\begin{cases}\n",
    "    1       & \\quad \\text{if } x > \\theta \\\\\n",
    "    0       & \\quad \\text{if } x \\leq \\theta\n",
    "  \\end{cases} \n",
    "$$\n",
    "\n",
    "This equation outputs a value of 1.0 for incoming values of $ \\theta$ (threshold value) or higher and 0 for all other values. Step functions, also known as threshold functions, only return 1 (true) for values that are above the specified threshold, as seen in Figure bellow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1eab92db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhhUlEQVR4nO3deZhcdZ3v8fc3nU7SnT0EAklYAoRg4ILQMSCLJmwJqIDXcAVHRAUzXIkz41UuMF588GFmBBkYNxYRGUAdMzOIGCECQdIoaDALkIUQCFkwnZCF7k7SW6qX7/2jqmOlUtVV1V2nzqmuz+t5+uk6dc6v6lOnu+pbZ/n9jrk7IiJSvgaEHUBERMKlQiAiUuZUCEREypwKgYhImVMhEBEpcwPDDpCvsWPH+jHHHNOrts3NzQwdOrSwgQogqrkgutmUKz/KlZ/+mGv58uW73P3QtDPdvaR+ampqvLcWL17c67ZBimou9+hmU678KFd++mMuYJln+FzVriERkTKnQiAiUuZUCEREypwKgYhImVMhEBEpc4GdPmpmDwMfB3a4+8lp5hvwPeASoAX4vLuvCCqPSFQ9+Woddz27jq2NrYwfVcWNs6Zw+WkT+m3b5PZ1ja1MWPJCSeQu1Gvuy3P3Zn3lIsh+BI8APwQeyzD/YmBy4ucM4P7Eb5Gy8eSrddzyxCpa2zsBqGts5ZYnVgFkfaOXYttSzV2qrzlXgRUCd/+9mR3TwyKXAY8lzm9dYmajzOwId98WVCaRqLnr2XX73+DdWts7ufXJ1XR0OXNqJgLw0B82sKe1/YDlfrpkc8a2G3Y2MXX8CGaffAQAT66PsSK2bv9y//7yph7bAnxo0hjOnXworbFO7q9dn1dbgBknHsbpR41m5959/PRPm/Ju//FTx3PCuOFs2tXMEyu25NUW4NPTj2LCqCpW1+3muTXvZWz7z0+vPajtps0xTpvezsjqSl7Z8D63Prk6bdu7nl3H2GGD+fPG90n1d+dPZmDFAF54c3uP7YdUVvDG1t0HzBtcWcENM48H4Fu/WZOxbaEKgXmA1yNIFIKnMuwaegq4w91fSkz/DrjJ3ZelWXYuMBdg3LhxNfPnz+9VnqamJoYNG9artkGKai6Ibrb+kuvzzzRnnHfSIQO48UNVANz4Ygu7Wg98r/b0zjXgrPED+dIpgwG47tkmOt1ybgvwsWMrmXPCIJpizldeaMmrLcBnThzEhcdUsmVvF7e+3Jp3+y9/cDAfOnwgq3d1cveytrzaAvzjGUOYPLqCl+ra+cmqWF5twfnOR6o5tHoAT70T4/G329O0ivvEsZU8teHg+Q9eVE3lAOPna/exaHNHxvbnThjIS3UHzq+uhHvPj/cg7ul/5JHZufcynjlz5nJ3n5ZuXpiF4Gng2ymF4P+6+/KeHnPatGm+bNlBtSIntbW1zJgxo1dtgxTVXBDdbP0l19l3vEBdY+tB908YVcXLN59XsLapuYr1vIVuX4rrq5i5e2JmGQtBmGcNbQGOTJqeCGwNKYtIKG6cNYWqyooD7quqrODGWVP6Zdswn7scX3Ouwhx0bgEwz8zmEz9IvFvHB6TcXH7aBLbvaeOeRW8R6+jK62yS7mV6cyZKWG1T29c1tjKhBHIX8jX35bnzXV85yzQIUV9/gF8A24B24t/+rwWuB65PzDfgXuAdYBUwLZfH1aBzxRXVbP0p1x/e2ulH3/SUL3lnV+EDJfSn9VUM/TEXPQw6F+RZQ1dlme/ADUE9v0ipqG+JATBm6KCQk0i5Us9ikZA1JgrBaBUCCYkKgUjI6pvjhWBUVWXISaRcqRCIhGz4kEo+eOQoBlbo7SjhKLlLVYr0N9eeM4lrz5kUdgwpY/oKIiJS5lQIREL2v3+2nG8vXBt2DClj2jUkErKVW3ZTPUhvRQmPtghEQlbfHGN0tc4YkvCoEIiEqK29k9b2TvUhkFCpEIiEqEG9iiUCVAhEQtTZ5Zxz/FiOHlMddhQpYzpCJRKiiaOr+dl1ukKrhEtbBCIiZU6FQCRE/7n0Xc79zgvsbsl8KUSRoKkQiIRo2+42/lLfytDBFdkXFgmICoFIiBqaY4wYMlADzkmo9N8nEqKGlnadOiqhUyEQCVFDS4xR1SoEEi6dPioSotOPGo1Z2Cmk3KkQiIToqxeeEHYEEe0aEgmTu4cdQUSFQCQsbe2dTLn1GR55eWPYUaTMqRCIhKShJUaso4shlepDIOFSIRAJSX1zfORRnTUkYVMhEAlJQ3N8WAn1I5CwqRCIhOSv1yLQ1ckkXCoEIiGZMLqKq6YfyaHDhoQdRcqc+hGIhOT0o0Zz+lGjw44hoi0CkbC0tXfS1aV+BBI+FQKRkNz4+EouuOfFsGOIqBCIhKWhOcbIah0olvAFWgjMbLaZrTOz9WZ2c5r5I83sN2b2upmtMbMvBJlHJEoaWmKMUR8CiYDACoGZVQD3AhcDU4GrzGxqymI3AG+4+6nADOBuM9M7Q8pCQ3OM0epDIBEQ5BbBdGC9u29w9xgwH7gsZRkHhpuZAcOAeqAjwEwikVHfEmO0dg1JBFhQox+a2Rxgtrtfl5i+GjjD3eclLTMcWACcCAwHPu3uT6d5rLnAXIBx48bVzJ8/v1eZmpqaGDZsWK/aBimquSC62Uo9V5c7C95p5/hRFZw8Nvixhkp9fRVbf8w1c+bM5e4+Le1Mdw/kB7gCeChp+mrgBynLzAH+DTDgeGAjMKKnx62pqfHeWrx4ca/bBimqudyjm0258qNc+emPuYBlnuFzNchdQ1uAI5OmJwJbU5b5AvBEIuf6RCE4McBMIpHQ1t7Jjj1tdHR2hR1FJNBCsBSYbGaTEgeAryS+GyjZu8D5AGY2DpgCbAgwk0gkLNvUwPR/+R0r3m0MO4pIcENMuHuHmc0DngUqgIfdfY2ZXZ+Y/wBwO/CIma0ivnvoJnffFVQmkaio14BzEiGBjjXk7guBhSn3PZB0eytwUZAZRKKoQdcikAhRz2KREOy/KE2VtggkfCoEIiFobIkxsqqSgRV6C0r4NAy1SAhmnXQ4x48bHnYMEUCFQCQUZx0/lrOOHxt2DBFAu4ZEQvH29r3s2NsWdgwRQIVAJBTXPPxn7vzturBjiAAqBCKhqG+JqQ+BRIYKgUiRtcY6aWvv0hDUEhkqBCJF1pDoVTxanckkIlQIRIqsuzOZCoFEhQqBSJGNH1XF3VecyqlHjgw7igigfgQiRTdm6CA+VTMx7Bgi+2mLQKTI/lLfwrJN9XR2BXN1QJF8qRCIFNkvV2xhzgN/6r5Kn0joVAhEiqyhOcaIIQM14JxEhv4TRYqsvqWdMepDIBGiQiBSZI0tMXUmk0hRIRApsvrmGGPUh0AiRKePihTZty49iYoBFnYMkf1UCESKbNoxY8KOIHIA7RoSKaJYRxdPr9zGloaWsKOI7KdCIFJEu5r2ccN/rOClt3eFHUVkPxUCkSLaP+CczhqSCFEhECmi7iGo1Y9AokSFQKSIGlraARhdrauTSXSoEIgUUYOuRSARpNNHRYroY6ccwQeOGMEoFQKJEBUCkSIaO2wwY4cNDjuGyAG0a0ikiBav28Hzb2wPO4bIAbRFIFJEP/nDRlpiHVwwdVzYUUT2C3SLwMxmm9k6M1tvZjdnWGaGmb1mZmvM7MUg84iErb45pgPFEjmBbRGYWQVwL3AhsAVYamYL3P2NpGVGAfcBs939XTM7LKg8IlHQ0BJj6vgRYccQOUCQWwTTgfXuvsHdY8B84LKUZT4DPOHu7wK4+44A84iErqElps5kEjkW1HVTzWwO8W/61yWmrwbOcPd5Sct8F6gETgKGA99z98fSPNZcYC7AuHHjaubPn9+rTE1NTQwbNqxXbYMU1VwQ3WylmGtfp/O3i1qYc0IlHz+2uMWgFNdXmPpjrpkzZy5392lpZ7p7ID/AFcBDSdNXAz9IWeaHwBJgKDAWeBs4oafHramp8d5avHhxr9sGKaq53KObrRRzdXZ2+eZdzb5rb1vxAiWU4voKU3/MBSzzDJ+rQZ41tAU4Mml6IrA1zTK73L0ZaDaz3wOnAm8FmEskFAMGGEcdUh12DJGD5HSMwMwGmNlpZvYxMzvPzHI5920pMNnMJpnZIOBKYEHKMr8GzjWzgWZWDZwBrM3nBYiUig07m3jgxXfYuXdf2FFEDtDjFoGZHQfcBFxAfLfNTmAIcIKZtQA/Ah51967Utu7eYWbzgGeBCuBhd19jZtcn5j/g7mvN7BlgJdBFfFfS6sK9PJHoWFW3mzt++yYXfOAwDh2u3sUSHdl2Df0TcD/wt4l9TPslTvX8DPF9/4+ma+zuC4GFKfc9kDJ9F3BXfrFFSk/j/pFHddaQREuPhcDdr+ph3g7gu4UOJNJf1TfHMIORVRqCWqKlx2MEZva1RMew1PsPMbOfBBdLpP9paIkxYkglAys0xJdES7b/yCnAcjM7u/sOM/sysAxYFWQwkf6moaVdnckkkrLtGpprZmcBPzSzNcCJxA8an+Xu24oRUKS/uGvOKTTv6wg7hshBculHsJr4qaCzAQO+piIgkr8hlRUMqTxoT6tI6LIdI/gs8BqwATgO+CTwHTN7TAPEieTnnkVvsUjXIpAIynaM4Apgprvf6e4d7r4c+DDwJ+JDQ4hIjh78/Tss3VQfdgyRg2Q7RpA6WiiJ/gT3m9njgaUS6WdaY520tXcxqlqnjkr0ZN01ZGZpl3H3nWZ2nJmdE0w0kf6jviUGwBh1JpMIynaw+BDgVTNbDiznr0NMHA98FNgFpL3ymIj8VUNzvBCM1umjEkHZdg19z8x+CJwHnA2cArQSHxjuak9cUEZEeranrR0zDS8h0ZT19FF37zSzV919UTECifRHZx03lrf/6WLMLOwoIgfJdozgE2a2E1hlZlsSnctEpBcGVgygYoAKgURPttNH/xk4192PAD4FfDv4SCL9z9Mrt/H/nlxFyiC+IpGQrRB0uPubAO7+CvHrCotInl7Z+D6/eX2bdg1JJGU7RnCYmf2fTNPufk8wsUT6Fw04J1GWrRD8mAO3AlKnRSQHDc0xRqszmURUttNHv1WsICL9WX1zjPGjhoQdQyStbGcN/VfS7TtT5j0XVCiR/mZghXHocBUCiaZsu4YmJ92+kPiF7LsdWvg4Iv3TgnkaiUWiK9tZQz2d66bz4ERE+oFshaDazE4zsxqgKnH79O7pIuQTKXk79rZx3aNLeWXD+2FHEUkr266hbUD3KaLvJd3unhaRLHbs2cfza3dwxbQjw44ikla2QnCLu+sCNCJ9UJ8YeVT9CCSqsu0auq8oKUT6sYbEtQg08qhEVbZCoP7wIn3UoC0Cibhsu4YmmdmCTDPd/dIC5xHpdwZWDGDi6CpGVqlnsURTtkKwE7i7GEFE+qvPnnk0nz3z6LBjiGSUrRA0ufuLRUkiIiKhyHaMoMHMDu+eMLPPmdmvzez7ZjYm4Gwi/cKtT67m9qfeCDuGSEbZCsEoIAZgZh8B7gAeA3YDDwaaTKSfWL65gc3vN4cdQySjbIVggLvXJ25/GnjQ3X/p7rcCx2d7cDObbWbrzGy9md3cw3IfMrNOM5uTe3SR0tDQEtOpoxJp2QrBQDPrPo5wPvBC8ryeGppZBXAvcDEwFbjKzKZmWO5O4NlcQ4uUCnenvjnGaJ06KhGWrRD8AnjRzH4NtAJ/ADCz44nvHurJdGC9u29w9xgwH7gszXJfAX4J7MgnuEgpaG3vZF9Hl7YIJNIs28W0zexM4AjgOXdvTtx3AjDM3Vf00G4OMNvdr0tMXw2c4e7zkpaZAPwHcB7wE+Apd388zWPNBeYCjBs3rmb+/Pl5vchuTU1NDBs2rFdtgxTVXBDdbKWSa2/M+d6KNmYdU8mHDs92kl7xckWFcuWnL7lmzpy53N2npZ3p7oH8AFcADyVNXw38IGWZ/wbOTNx+BJiT7XFramq8txYvXtzrtkGKai736GZTrvwoV376Yy5gmWf4XA3yK8oWIHm4xYnA1pRlpgHzzQxgLHCJmXW4+5MB5hIRkSTZjhH0xVJgsplNMrNBwJXAAcNVuPskdz/G3Y8BHge+rCIg/clza95j1r/9nrrG1rCjiGQU2BaBu3eY2TziZwNVAA+7+xozuz4x/4GgnlskKrY2trJu+16GDAzyO5dI3wR69MrdFwILU+5LWwDc/fNBZhEJQ31LO2ZowDmJNH1NEQlQY0uMkVWVDKzQW02iS/+dIgGqb1avYom+8E5sFikDJ4wbrgvSSOSpEIgE6O/Onxx2BJGstGtIRKTMqRCIBOiMf3me+2rXhx1DpEcqBCIBaY11sn3PPgwLO4pIj1QIRAJS3xIDYHS1+hBItKkQiASkoTlRCHTWkEScCoFIQBoSWwQ6fVSiToVAJCCjqwdx+QfHM35UVdhRRHqkfgQiATl5wki+e+VpYccQyUpbBCIB6erq+ep/IlGhQiASkNt+s4azvv27sGOIZKVCIBKQ+uYYQyorwo4hkpUKgUhAGlpiOnVUSoIKgUhA6pvbNQS1lAQVApGANDTH1KtYSoJOHxUJyBXTJnLi4SPCjiGSlQqBSEC+dtGUsCOI5ES7hkQC0N7ZxZ62dtzVl0CiT4VAJABvbtvLKbc9x6I3tocdRSQrFQKRAGjAOSklKgQiAeguBKN0+qiUABUCkQDUN2uLQEqHCoFIABpa2jGDkVXqRyDRp9NHRQJw5rFjGFRxAhUDdL1iiT4VApEAnHXcWM46bmzYMURyol1DIgGoa2ylMXHAWCTqVAhEAnDdo8v4+n+/HnYMkZwEWgjMbLaZrTOz9WZ2c5r5f2NmKxM/fzSzU4PMI1Is8QHndMaQlIbACoGZVQD3AhcDU4GrzGxqymIbgY+6+ynA7cCDQeURKRZ3p6ElplNHpWQEuUUwHVjv7hvcPQbMBy5LXsDd/+juDYnJJcDEAPOIFEVreyf7Orp0URopGRbUoFhmNgeY7e7XJaavBs5w93kZlv86cGL38inz5gJzAcaNG1czf/78XmVqampi2LBhvWobpKjmguhmi3Kutopqvv5iK184eRAfnRiNfgRRXl/Klbu+5Jo5c+Zyd5+Wdqa7B/IDXAE8lDR9NfCDDMvOBNYCh2R73JqaGu+txYsX97ptkKKayz262aKcq7El5j/90yZfv2Nv2HH2i/L6iqL+mAtY5hk+V4PsR7AFODJpeiKwNXUhMzsFeAi42N3fDzCPSFGMrKrks2ceHXYMkZwFeYxgKTDZzCaZ2SDgSmBB8gJmdhTwBHC1u78VYBaRotmxt401W3fT3tkVdhSRnARWCNy9A5gHPEt8t89/ufsaM7vezK5PLPZN4BDgPjN7zcyWBZVHpFgWrtzGx77/EnvbOsKOIpKTQIeYcPeFwMKU+x5Iun0dcNDBYZFSVq8B56TEqGexSIE1NMcYWVWpAeekZKgQiBRYfUuMMepVLCVEhUCkwBpbYupMJiVFw1CLFNhXzptMR2cwHTVFgqBCIFJgZx57SNgRRPKiXUMiBeTuLH5zB1saWsKOIpIzFQKRAop1whceWcpTK7eFHUUkZyoEIgW0tz1+bGB0tfoQSOlQIRApoKZYdyHQWUNSOlQIRAqoKbFFoIvSSClRIRApoL2J69WrH4GUEhUCkQKaekgFj35xOhNGVYUdRSRn6kcgUkAjBxsfPeHQsGOI5EVbBCIF9HZDJ4vf3BF2DJG8qBCIFNCize3c/vQbYccQyYsKgUgB7Y25Th2VkqNCIFJATe3qQyClR4VApICaYs6YoepVLKVFhUCkQNydve2uPgRScnT6qEgB3fbhKs778DFhxxDJi7YIRArEzJg4fADj1ZlMSowKgUiBvLe7jec3t/Pe7rawo4jkRYVApEDWbd/Lz9bGdFEaKTkqBCIF0tgSH3FOB4ul1KgQiBRIfXO8EIxRPwIpMSoEIgXS0BzDgBFV6kcgpUWFQKRA6ltiDK2EigEWdhSRvKgfgUiB3DT7RE4dtDPsGCJ50xaBSIEMH1LJodV6S0np0X+tSIH8+8sbWbG9I+wYInnTriGRAvnRixuYPLwz7BgieQu0EJjZbOB7QAXwkLvfkTLfEvMvAVqAz7v7ikLnePLVOu56dh11ja1MWPICN86awuWnTci7/dbGVsaPqsqrfSm2TW7fm3VWiq+5r+vrVyu28N6eNt7bA2ffkf//mEiYAisEZlYB3AtcCGwBlprZAndPvnzTxcDkxM8ZwP2J3wXz5Kt13PLEKlrb49/U6hpbueWJVQA5vVH70r4U25Zq7tDX169W7Z/Ot71I2Mzdg3lgsw8Dt7n7rMT0LQDu/u2kZX4E1Lr7LxLT64AZ7r4t0+NOmzbNly1blnOOs+94gbrG1oPuHzjAmDR2KJUVA1j49+cCcOczb/L8G9sPWG7z+y3EOrsytj985BB+em28dt30+EpWvNuwf5mNu5rp6Dp4/Xa3BThp/AguP3w3M2bM4LpHl7L5/Zac2wKcO/lQvvmJqQB88r6XaWrryLktxD+obph5PPs6Ovn491/K67kBrj1nEldOP4qtja1c8/CfM7Y9bPhgRqY5v/7GWVO46KTDWV23m8vvfTlt2wmjqvjulR/kH5/464dtc0szQ6uHcsen/gc1R4+h5vZFvJ/o0JX6vH/+xgU8s3obdz/31kHzf/y5afzNQ69k/B/5483ncdiIIfz8lc088vKmg5b55ZfP4uLv/iFt+wmjqnj55vMOuj8MtbW1zJgxI+wYB1Gu/PQll5ktd/dp6eYFuWtoAvCXpOktHPxtP90yE4ADCoGZzQXmAowbN47a2tqcQ6R7gwJ0dDkjrZWBsP/x9mxvZ6QduI83XRFIbl+5r21/+1hjjJHWdcAyPbUF6Nyzj6ZhMWpraxnQso+R5jm3BWjeVUdtbfxi6dUd+6gwz7ktQH3dRmprtyTm7cvruQHqNr5FbcsGdu+Lt8/UdsfefRxVffCB1A3r1lC78022NnVlbFvX2MobK19lpLXvv2/Y4C4qrJU1r7/K3o0VaYtA9/PW1tay4f3OA9p3W740fRHofr1LlvyJEYOMre91MNIOzv/Hl17K2L6usTWv/9UgNTU1RSZLMuXKT1C5giwE6XrVpL7Tc1kGd38QeBDiWwT5VMQJS9JvEUwYVcXjXz3w21q6R820RZG2fcoD5Nq2u8ont8/neVOfvzdtAS44r2/tL5vV+7YA967O3PaaS8/jmqT7Ur8Z9fR3njFjBjOAL2d43ntez9z20otmAun/N/Yvt6zn546C/vgNN0jllivI00e3AEcmTU8EtvZimT65cdYUqiorDrivqrKCG2dNCbx9KbYN87lLsW0h2ouELcgtgqXAZDObBNQBVwKfSVlmATDPzOYT3220u6fjA73RfbBu/xkweZ4Rktw+3zNKSrFtavt811kpvuYw15dIJLh7YD/ETwt9C3gH+EbivuuB6xO3jfiZRe8Aq4Bp2R6zpqbGe2vx4sW9bhukqOZyj2425cqPcuWnP+YClnmGz9VA+xG4+0JgYcp9DyTdduCGIDOIiEjPNMSEiEiZUyEQESlzKgQiImVOhUBEpMwFNsREUMxsJ7C5l83HArsKGKdQopoLoptNufKjXPnpj7mOdvdD080ouULQF2a2zDOMtRGmqOaC6GZTrvwoV37KLZd2DYmIlDkVAhGRMlduheDBsANkENVcEN1sypUf5cpPWeUqq2MEIiJysHLbIhARkRQqBCIiZa7fFQIzu8LM1phZl5lNS5l3i5mtN7N1ZjYrQ/sxZrbIzN5O/B4dQMb/NLPXEj+bzOy1DMttMrNVieVyvz5n73PdZmZ1SdkuybDc7MQ6XG9mNxch111m9qaZrTSzX5nZqAzLFWV9ZXv9Fvf9xPyVZnZ6UFmSnvNIM1tsZmsT//9/n2aZGWa2O+nv+82gcyU9d49/m5DW2ZSkdfGame0xs39IWaYo68zMHjazHWa2Oum+nD6LCvJ+zDQsaan+AB8ApgC1JA1rDUwFXgcGA5OID31dkab9d4CbE7dvBu4MOO/dwDczzNsEjC3iursN+HqWZSoS6+5YYFBinU4NONdFwMDE7Tsz/U2Ksb5yef3Eh1//LfFh1s8EXinC3+4I4PTE7eHEh39PzTUDeKpY/0/5/G3CWGdp/q7vEe90VfR1BnwEOB1YnXRf1s+iQr0f+90Wgbuvdfd1aWZdBsx3933uvhFYD0zPsNyjiduPApcHEpT4tyDgfwG/COo5AjAdWO/uG9w9Bswnvs4C4+7PuXv3BYOXEL+SXVhyef2XAY953BJglJkdEWQod9/m7isSt/cCa4lf/7tUFH2dpTgfeMfdeztqQZ+4+++B+pS7c/ksKsj7sd8Vgh5MAP6SNL2F9G+UcZ64Slri92EBZjoX2O7ub2eY78BzZrbczOYGmCPZvMSm+cMZNkVzXY9B+SLxb47pFGN95fL6Q11HZnYMcBrwSprZHzaz183st2Z2UrEykf1vE/b/1ZVk/kIW1jrL5bOoIOst0AvTBMXMngcOTzPrG+7+60zN0twX2LmzOWa8ip63Bs52961mdhiwyMzeTHxzCCQXcD9wO/H1cjvx3VZfTH2ING37vB5zWV9m9g2gA/h5hocp+PpKFzXNfamvv6j/awc8sdkw4JfAP7j7npTZK4jv+mhKHP95EphcjFxk/9uEuc4GAZcCt6SZHeY6y0VB1ltJFgJ3v6AXzbYARyZNTwS2plluu5kd4e7bEpumO4LIaGYDgf8J1PTwGFsTv3eY2a+Ibwb26YMt13VnZj8GnkozK9f1WNBcZnYN8HHgfE/sHE3zGAVfX2nk8voDWUfZmFkl8SLwc3d/InV+cmFw94Vmdp+ZjXX3wAdXy+FvE8o6S7gYWOHu21NnhLnOyO2zqCDrrZx2DS0ArjSzwWY2iXhV/3OG5a5J3L4GyLSF0VcXAG+6+5Z0M81sqJkN775N/IDp6nTLFkrKPtlPZni+pcBkM5uU+CZ1JfF1FmSu2cBNwKXu3pJhmWKtr1xe/wLgc4kzYc4Edndv4gclcbzpJ8Bad78nwzKHJ5bDzKYTf/+/H2SuxHPl8rcp+jpLknHLPKx1lpDLZ1Fh3o9BHw0v9g/xD7AtwD5gO/Bs0rxvED/Cvg64OOn+h0icYQQcAvwOeDvxe0xAOR8Brk+5bzywMHH7WOJnALwOrCG+iyTodfdTYBWwMvHPdERqrsT0JcTPSnmnSLnWE98P+lri54Ew11e61w9c3/33JL65fm9i/iqSzl4LMNM5xHcJrExaT5ek5JqXWDevEz/oflbQuXr624S9zhLPW038g31k0n1FX2fEC9E2oD3x+XVtps+iIN6PGmJCRKTMldOuIRERSUOFQESkzKkQiIiUORUCEZEyp0IgIlLmVAhE+sDio35uNLMxienRiemjw84mkisVApE+cPe/EB+a447EXXcAD3pIg5eJ9Ib6EYj0UWJoh+XAw8CXgNM8PhKkSEkoybGGRKLE3dvN7EbgGeAiFQEpNdo1JFIYFxMfIuDksIOI5EuFQKSPzOyDwIXEr6z11SJfUEWkz1QIRPogMTLl/cTH/38XuAv413BTieRHhUCkb74EvOvuixLT9wEnmtlHQ8wkkhedNSQiUua0RSAiUuZUCEREypwKgYhImVMhEBEpcyoEIiJlToVARKTMqRCIiJS5/w9DrKTEfxD9fAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# def step(x):\n",
    "#     if x > 0.5:\n",
    "#         return 1\n",
    "#     else:\n",
    "#         return 0\n",
    "\n",
    "# # y = []\n",
    "# # for i in x:\n",
    "# #     y.append(step(i))\n",
    "# y = list(map(step, x))\n",
    "\n",
    "y = list(map(lambda n: 1 if n>0.5 else 0, x))\n",
    "plot_graph(y,\"STEP(X)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f7f7c24",
   "metadata": {},
   "source": [
    "### Sigmoid or Logistic Activation Function\n",
    "\n",
    "\n",
    "The sigmoid non-linearity has the mathematical form as bellow.\n",
    "\n",
    "$$ \\sigma (x) =  \\frac{1}{1 + e^ \\left(-x \\right)} $$ \n",
    "\n",
    "it takes a real-valued number and **squashes** it into range between __0 and 1__. Therefore, it is especially used for models where we have to predict the probability as an output. Since probability of anything exists only between the range of 0 and 1, sigmoid is the good choice.\n",
    "\n",
    "In particular, large negative numbers become 0 and large positive numbers become 1. The sigmoid function has seen frequent use historically. The sigmoid non-linearity has recently fallen out of favor and it is rarely ever used - because of the following drawbacks:\n",
    "\n",
    "- Vanishing gradient—for very high or very low values of X, there is almost no change to the prediction, causing a vanishing gradient problem. This can result in the network refusing to learn further, or being too slow to reach an accurate prediction.\n",
    "- Outputs not zero centered.\n",
    "- Computationally expensive\n",
    "\n",
    "\n",
    "The Sigmoid Function curve looks like a S-shape.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "383669a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnkElEQVR4nO3de3xU9Z3/8dcnCYFAkCBohIACFbEICgZB8VLwBlq3WhddbWtbW0tty6+1q1Tttt1t3a1U1l60Vupa13a7K8WWorUoWgEvrVRBbiIEEeQSkKtcArlN8vn9MRMMYYZMLidnLu/n4zEPMuec7+Sdk3A+c77ne75j7o6IiGSvnLADiIhIuFQIRESynAqBiEiWUyEQEclyKgQiIlkuL+wALdW7d28fMGBAq9oePHiQbt26tW+gdpCquSB1sylXyyhXy2RiriVLluxy9xPirnT3tHqUlpZ6ay1YsKDVbYOUqrncUzebcrWMcrVMJuYCFnuC46q6hkREspwKgYhIllMhEBHJcioEIiJZToVARCTLBTZ81MweA64Cdrj7sDjrDfgZcCVwCPi8u78ZVB4RkTlLy5k+r4yteyvpW1TA1AlDuGZkSeBt2+t7l++tpGTR/BZ/7+YEeR/B48DPgd8kWH8FMDj2GAM8HPtXRCSh1h5Q5ywt5+7ZK6msrQOgfG8ld89eCdBs+7a0Dft7JyOwQuDuL5vZgGNscjXwm9j41kVmVmRmfdx9W1CZRCQ1tPYdbryD4l2zV7CvsobPnjcAM2P7/ip2V9RQW1dPpL6e2jqn3p3p88oOt2tQWVvH9/+0itwc4x/O6gvAy2t3smBTLZsXbQR36h1+8pe1cdt+76m32Lavil7d8rn+nP4A/H7JFrbvr4qN0Y9u+6u/bojb/rtPvcV7uw9ySq+ufHJkPwAefWU9+6sih7f77wRtp88ra7dCYB7g5xHECsEzCbqGngGmufursecvAne6++I4204GJgMUFxeXzpw5s1V5KioqKCwsbFXbIKVqLkjdbMrVMqmU629ba3n8rRpq6j9clmfw2aGduKh/Pm/vruPN7REqI1BV51RGnMoIfOucLnzn1Up2V8U/Zj16eVfycoz/ebuaFzdFjliXZxA5xqGusBP8/JLoHbsPLq1iyfa6xBvH0a/Q+PcLugLwg9cqWb+vvpkWRxrWK5c7zukCwO0LDyX8GZt6fGLydxmPHz9+ibuPircuzCkmLM6yuD+9uz8CPAIwatQoHzduXKu+4cKFC2lt2yClai5I3WzK1TLtnSuZ7pnauno27znEe7sPsn7nQT4xoi8ndu/Cbd9//ogiANGD9DObcvjeTeNY/+oGXl+5lu5dOlHYOY/CwjxO6JzHeWNHsOcvLyTMNO5jHyMvN4feg/dx3QeHyMvJIS/XyM/NIS83h2/+bhnleyuPald8XGd+f+tY+h8fPZCfdU4NL73yKmPPH0uOGTlmfPyBV9i2r+qotn16dGH+7eMwgy6dcgEYe8GHP5xZ9EB30fQFbN17dPuSoi68eufFsW2jh8Ql447c5vxp8+PmLikqaLffaZiFYAvQv9HzfsDWkLKISJLidc986/cr2FtZw+fHDmTxe3u448nlbP6gkrr6D9/bnXpiIScO6cK+ytq4r7vzQDUAX7hgIF+4YGDcbfoWFSQ8KOblRgdBDivpwbCSHkdtM3XCkCNyAxR0yuXuKz56uAgA9OyWT1GXHE7s3uXwsjsnnh637Z0TT6cgP/eI75Ofd/RgzG9NiN9+6oTTDxeARBLlnjphyDHbtUSYheBpYIqZzSR6kXifrg+IpL775q05qs+6pq6en77wDp8fO5Dju+VzRt8eXHVmXwb07sbA2KNn105A4oN536KCZr93Ww6KDWcsrbnQ3Ja27fm9y/dWUtKKEUvNCXL46BPAOKC3mW0B/hXoBODuM4C5RIeOriM6fPTmoLKISNvVROrJz8thW5wuDuDwO/1BJxTy0KfPTvg6YR3MG9q39gDalrbt9b2D6noMctTQjc2sd+BrQX1/EWm7qto65q16nycXb6G2rp7fffm8Nr2jh7a/w23rAVmOlnafRyAiwSt7/wC/XbSRp5aVs78qQr+eBVxX2p/6em+XPuug3+FKy6gQiGSppiN/vjJuENee3Y+u+Xn8fcNuZi3ezBXDTuL6Uf05d1AvcnKiFzXb2j0jqUeFQCQLxRv58505q1ixZR/3TTqLSaX9uHpECT0KOsVtr+6ZzKJCIJKF4t1lC7CwbCcAXfN1aMgmmn1UJAttjXOxFz4cyy/ZRYVAJAslGuGT7MgfySwqBCJZZNu+Su5/vow7LjuNgk5H3hHb3nerSvpQIRDJEht2HWTSw6/x+F/fY8QpPbn32uGUFBVgRKdouPfa4boAnKV0RUgkC2zcX8cdM/6GOzwx+dzD0z7owC+gMwKRjLf4vT1Me72K/NwcZt16XtwJ2SS76YxAJMNVR+rpXZDDzK+MpUQXgyUOnRGIZKiNuw8CcP6pvfn+2C4qApKQCoFIBnri9U1cfP9LvLh6OwA5zcx5L9lNXUMiGWbGS+8y7dk1jBtyAmM/0jvsOJIGVAhEMoS7c9+8Mh5e+C5XndmHH18/Iu6nZYk0pUIgksYazyDaqzCfXRU1fGrMydxz9TByc9QdJMlRIRBJU01nEN1VUUN+bg7nnNJTRUBaROeNImkq3gyiNXX1/Ofza0NKJOlKhUAkTSWaQTTRcpFEVAhE0pRmEJX2okIgkqZuOKf/Ucs0g6i0hgqBSJpasukDCjrl0KdHF80gKm2iUUMiacjdueSjxUw44yRuHH1y2HEkzakQiKQhM+Omc08JO4ZkCHUNiaSZN97bw28XbaS2rj7sKJIhVAhE0oi7M+3ZNfx8/jrq6j3sOJIhVAhE0shLa3eyZOMHTLn4VLo0+cxhkdZSIRBJE+7O/c+vpV/PAq4fdfTQUZHWUiEQSRPPv72dleX7+PolgzWrqLQr/TWJpIkeBZ24cvhJXKv7BKSdafioSJo4d1Avzh3UK+wYkoECPSMws4lmVmZm68zsrjjre5jZn8xsuZmtMrObg8wjko4idfXMeOld9h6qCTuKZKjACoGZ5QIPAVcAQ4EbzWxok82+Brzt7mcB44D7zSw/qEwi6eipZVuZ9uwaXnt3d9hRJEMFeUYwGljn7uvdvQaYCVzdZBsHupuZAYXAHiASYCaRtFJbV8/PXnyHM/oex8RhJ4UdRzKUuQdzU4qZTQImuvstsec3AWPcfUqjbboDTwOnA92Bf3L3P8d5rcnAZIDi4uLSmTNntipTRUUFhYWFrWobpFTNBambLVtyLdxcy+Orarjt7M6MOLH1l/SyZX+1l0zMNX78+CXuPiruSncP5AFcBzza6PlNwINNtpkE/AQw4FRgA3DcsV63tLTUW2vBggWtbhukVM3lnrrZsiFXVW3Ez/vhX/yah171+vr6Nr1WNuyv9pSJuYDFnuC4GmTX0Bag8V0v/YCtTba5GZgdy7kuVghODzCTSNrYXxlhaN8e3H7ZEKK9pyLBCHL46BvAYDMbCJQDNwCfarLNJuAS4BUzKwaGAOsDzCSSNk7o3plHPxf/TF6kPQV2RuDuEWAKMA9YDcxy91VmdquZ3Rrb7B5grJmtBF4E7nT3XUFlEkkXC9bsYMOug2HHkCwR6A1l7j4XmNtk2YxGX28FLg8yg0i6qaiOcPuTyzmzXw8ev3l02HEkC2iKCZEU8/hfN7DnYA23XXpa2FEkS6gQiKSQfZW1PPLyei796ImM6F8UdhzJEioEIinkV6+sZ39VhG9eprMB6TgqBCIpJFLvXDOiL2f07RF2FMkimn1UJGRzlpYzfV4ZW/dW0reogDsu19mAdCwVApEQzVlazt2zV1JZWwdA+d5Kvv3HtzAzrtHnDkgHUdeQSIimzys7XAQaVNbWMX1eWUiJJBupEIiEaOveyhYtFwmCCoFIiPoWFbRouUgQVAhEQjR1whBymswnV9Apl6kThoQTSLKSCoFIiK46sw/du+TRLT8XA0qKCrj32uG6UCwdSqOGREKUl5vD4u9cxqHqOnp07RR2HMlSKgQiIeuUm0OPrjo5l/Dor08kJAerI0z4ycu88Pb2sKNIllMhEAnJ/DU7KNt+gO5ddGIu4VIhEAnJ3JXbOKF7Z84ZcHzYUSTLqRCIhOBgdYT5a3ZwxbCTyG06flSkg6kQiIRg/podVEfquXJ4n7CjiKgQiIShb1EBN5zTX91CkhJ0lUokBKWn9KT0lJ5hxxABdEYg0uHWbj/A+p0VYccQOUyFQKSD/eSFtVz/y0XU13vYUUQAFQKRDtUwWujK4SeRo9FCkiJUCEQ6kEYLSSpSIRDpQLqJTFKRCoFIB6mJ1PPqul26iUxSjoaPinSQ/LwcXr3zYqojdc1vLNKBVAhEOlCPgk6APndAUkvSXUNm1tPMzjCzQWamLiWRFjhUE+HGRxbxt3d3hR1F5CjHPKCbWQ8z+7aZrQQWAb8EZgEbzexJMxvfESFF0t38NTt4bf1uckzXBiT1NPfO/vfAZuBCdx/i7he4+yh37w9MA642sy8mamxmE82szMzWmdldCbYZZ2bLzGyVmb3U6p9EJIX9eYVGC0nqOuY1Ane/7BjrlgBLEq03s1zgIeAyYAvwhpk97e5vN9qmCPgFMNHdN5nZiS2LL5L6DtVEWFC2g+tH9ddoIUlJzXUN/WOC5flm9t1mXns0sM7d17t7DTATuLrJNp8CZrv7JgB335FcbJH0MX/NDqpqdROZpC5zTzzfiZnNA+qBr7r7htiyK4CfAM+5+23HaDuJ6Dv9W2LPbwLGuPuURtv8lOgQijOA7sDP3P03cV5rMjAZoLi4uHTmzJkt+yljKioqKCwsbFXbIKVqLkjdbOmUa+XOCAu3RPjaiM6hXSNIp/2VCjIx1/jx45e4+6i4K939mA/gRuBd4B7gj8CrwFlJtLsOeLTR85uAB5ts83OiF6G7Ab2Bd4DTjvW6paWl3loLFixoddsgpWou99TNplwto1wtk4m5gMWe4LiazH0Es4i+Y/8msBe42N3XJtFuC9C/0fN+wNY42+xy94PAQTN7GTgLSOb1RVLept2H6N4lj57d8sOOIpJQc9cILgCWAr2IHtSnAH8ysx+YWedmXvsNYLCZDTSzfOAG4Okm2zwFXGhmeWbWFRgDrG7FzyGSkqY9t5orH3il4QxYJCU1N3z0p8At7v4Vd//A3ecAI4HOwPJjNXT3CNHCMY/owX2Wu68ys1vN7NbYNquB54AVwOtEu5LeasPPI5IyDtVEp5y+9KPFmO4fkBTWXNfQaHevb7zA3Q8Bd5rZ4829uLvPBeY2WTajyfPpwPSk0oqkkYbRQh8/U6OFJLU1d0YwNtEKd19tZseZ2bB2ziSSEeau3EbvQt1EJqmvuTOCfzSz+4h23ywBdgJdgFOB8cApwO2BJhRJQ1W1dSxYs5NJpf10E5mkvObuLP6mmfUEJhEdDtoHqCTa5/9Ld381+Igi6adLp1zm3XYRujQg6aDZ4aPu/gHwX7GHiCTp5F5dw44gkpRjFgIz++djrXf3H7dvHJH0d6gmwrd+v4LJFw3izH5FYccRaVZzF4u7xx6jgK8AJbHHrcDQYKOJpKf5a3bwzIptHKzWJ5FJemjuGsH3AczseeBsdz8Qe/5vwJOBpxNJQw2jhUYP1GghSQ/JftLYyUBNo+c1wIB2TyOS5qojzvw1O/QB9ZJWkv3M4v8BXjezPwIOfBI4apZQkWy3fGedppyWtJNUIXD3/zCzZ4ELY4tudvelwcUSSV+jBxyvbiFJK82NGjrO3feb2fHAe7FHw7rj3X1PsPFE0svoPnl868bzwo4h0iLNnRH8H3AV0buKHWjc6enAoIByiaSVOUvLmfbsGt7fX0XJovlMnTCEa0aWhB1LJCnNjRq6KvbvwI6JI5J+5iwt5+7ZK6msjQ4XLd9byd2zVwKoGEhaSHbUEGb2CTP7z9jjqiBDiaST6fPKDheBBpW1dUyfVxZSIpGWSaoQmNk04BvA27HHN8zs3iCDiaSLrXsrW7RcJNUkO3z0SmBEw2cTmNmviX5y2d1BBRNJF32LCiiPc9DvW1QQQhqRlku6awgoavR1j3bOIZK2vnHJqUctK+iUy9QJQ0JII9JyyZ4R3AssNbMFREcOXYTOBkQAmFTan/K9VTzx+iZ2HKimpKhAo4YkrSR7Q9kTZrYQOIdoIbjT3d8PMphIusjJMb552Wl887LTWLhwIePGjQs7kkiLtKRr6ITYv7nAWDO7NoA8ImnlUE2Ef33qLdbtqAg7ikirJXVGYGaPAWcCq4CGD7N3YHZAuUTSwoI1O/n1axuZOKwPp55YGHYckVZJ9hrBue6uzx8QaUJTTksmSLZr6DUzUyEQaeRQTYQX12xn4rBiTTktaS3ZM4JfEy0G7wPVRC8Yu7ufGVgykRS3YM1Oqmrr+fjwvmFHEWmTZAvBY8BNwEo+vEYgktX2V9VyWnGhuoUk7SVbCDa5+9OBJhFJMzeOPpkbzumPmbqFJL0lWwjWmNn/AX8i2jUEgLtr1JBkpQNVtRR2zlMRkIyQbCEoIFoALm+0TMNHJWvd9YeVbN1XyR+/en7YUUTaLNk7i28OOohIujhUE2H+mh38Y6mmkJDMkOwNZQ/EWbwPWOzuT7VvJJHUtmDNTipr6/QB9ZIxkr2PoAswAngn9jgTOB74opn9NJBkIikqehNZPmMG9go7iki7SLYQnApc7O4PuvuDwKXAR4FPcuR1gyOY2UQzKzOzdWZ21zG2O8fM6sxsUkvCi3S0D28iO0k3kUnGSPZicQnQjWh3ELGv+7p7nZlVx2tgZrnAQ8BlwBbgDTN72t3fjrPdj4B5rcgv0qHycnL46T+NZEDvrmFHEWk3yRaC+4BlsamoGz6P4Idm1g34S4I2o4F17r4ewMxmAlcT/ajLxv4f8AeiU1yLpLT8vBwmDjsp7Bgi7crcPbkNzfoQPbgb8Lq7b21m+0nARHe/Jfb8JmCMu09ptE0J8H/AxcCvgGfc/fdxXmsyMBmguLi4dObMmUllbqqiooLCwtSbITJVc0HqZgsjV3XEmbexlvP75tGrIH6vqvZXyyhXy7Ql1/jx45e4+6i4K9094QM4Pfbv2fEezbS9Dni00fObgAebbPMk0ZlNAR4HJh3rNd2d0tJSb60FCxa0um2QUjWXe+pmCyPXM8u3+il3PuN/Xbcz4TbaXy2jXC3TllxER3nGPa421zX0z0Tfid/fUDearL/4GG23AP0bPe8HND2LGAXMjN2d2Ru40swi7j6nmVwiHU6jhSRTNTdq6FEzO8ndx7v7eKKzkFYAbwHNjfB5AxhsZgPNLB+4AThiviJ3H+juA9x9APB74KsqApKKGm4i02ghyUTNFYIZQA2AmV1E9EPsf0109NAjx2ro7hFgCtHRQKuBWe6+ysxuNbNb2xpcpCPpJjLJZM11DeW6+57Y1/8EPOLufwD+YGbLmntxd58LzG2ybEaCbT/fbFqRkGz+4BAlRQXqFpKM1GwhMLO82Lv7S4iN3EmyrUjGuPVjH+GLFwxUt5BkpOYO5k8AL5nZLqASeAXAzE7lw5vLRDJapK6evNwcOuUmeyO+SHo5ZiFw9/8wsxeBPsDzsSFIEL228P+CDieSCm773TKqI/X812fjD8EWSXfNdu+4+6I4y9YGE0cktVTW1PHiak05LZlN57oix7CgbIdGC0nGUyEQOYY/6yYyyQIqBCIJVNbUMX+1biKTzKchoCIJOM63P/5RRvYvCjuKSKBUCEQS6Jqfx03nnhJ2DJHAqWtIJI7KmjqeeH0Tew/VhB1FJHAqBCJxLCzbwd2zV/L21v1hRxEJnAqBSBzPxEYLjR54fNhRRAKnQiDSRMNooQlnnESeppWQLKC/cpEmFsZuIvu4biKTLKFCINLEivJ96haSrKLhoyJN3DnxdL4y7iPqFpKsob90kUYaJtg9rkunkJOIdBydEYgAc5aWM31eGeV7K+man8sPPzmca0ZqxlHJDioEkvXmLC3n7tkrqaytA+BQTR13z14JoGIgWUFdQ5L1ps8rO1wEGlTW1jF9XllIiUQ6lgqBZL2teytbtFwk06gQSNbrW1TQouUimUaFQLLe1AlD6NLpyP8KBZ1ymTphSEiJRDqWLhZL1mu4IDx9Xhlb91bSt6iAqROG6EKxZA0VAslq1ZE6vvHEMr500SD+etfFYccRCYW6hiSrzXpjM8+tep+D1ZGwo4iERoVAslZVbR0Pzl/H6AHHc+Hg3mHHEQmNCoFkrd8u2siOA9XcfvlpmOnD6SV7qRBIVjpYHeHhhe9y4eDejBnUK+w4IqHSxWLJSnm5xpSLT+Xsk3uGHUUkdIGeEZjZRDMrM7N1ZnZXnPWfNrMVscffzOysIPOINOicl8vN5w/krP5FYUcRCV1ghcDMcoGHgCuAocCNZja0yWYbgI+5+5nAPcAjQeURafCb195j1hubw44hkjKCPCMYDaxz9/XuXgPMBK5uvIG7/83dP4g9XQT0CzCPCB8crOG+58qYv2ZH2FFEUoY1fBBHu7+w2SRgorvfEnt+EzDG3ack2P4O4PSG7ZusmwxMBiguLi6dOXNmqzJVVFRQWFjYqrZBStVckLrZWptrVlkNz26o5Z7zC+jXvf3fB2Xa/gqacrVMW3KNHz9+ibuPirvS3QN5ANcBjzZ6fhPwYIJtxwOrgV7NvW5paam31oIFC1rdNkipmss9dbO1JteO/VV++nee9a8/8Wb7B4rJpP3VEZSrZdqSC1jsCY6rQY4a2gL0b/S8H7C16UZmdibwKHCFu+8OMI9kuYcXvhudUuKSwWFHEUkpQRaCN4DBZjYQKAduAD7VeAMzOxmYDdzk7msDzCLC2I/0omfXTgw6IfVO+UXCFFghcPeImU0B5gG5wGPuvsrMbo2tnwF8D+gF/CJ2Z2fEE/VhibTRpUOLuXRocdgxRFJOoDeUuftcYG6TZTMafX0LcNTFYZH2tHnPIZ5csoUvXTiQ7l06hR1HJOVoignJeA/Of4cZL71LhWYYFYlLhUAy2oZdB/nDm+V8Zswp9Omhj54UiUeFQDLaz/6ylvzcHL4y7iNhRxFJWSoEkrHWbj/AU8u38rmxAzihe+ew44ikLBUCyVg5Zkw84yS+fNGgsKOIpDRNQy0Z69QTC3n4M6VhxxBJeTojkIz02Ksb2LT7UNgxRNKCzggkY8xZWs70eWVs3VuJA4vf28MvdEYg0iydEUhGmLO0nLtnr6Q8VgQA5pftYM7S8lBziaQDFQLJCNPnlVFZW3fEsqraeqbPKwspkUj6UCGQjLB1b2WLlovIh1QIJCP0LYp/13Ci5SLyIRUCSXt/XbeLyRcNpKBT7hHLCzrlMnXCkJBSiaQPFQJJa39esY3P//frLNm4l3uvHU5JUQEGlBQVcO+1w7lmZEnYEUVSnoaPStqa+fomvv3HlZSe0pN7rhlGj4JOOvCLtIIKgaSluRtqmFW2ko+ddgIzPlNKQX5u841EJC4VAkk7VbV1/K08wlVn9uHH148gP089nCJtoUIgaaOu3qmrd7p0yuWu0QVccelIcnMs7FgiaU9vpSQt1NbVc9vvlvH1J5ZSX+8U5puKgEg7USGQlFdZU8fk3yzmT8u3clb/InJUAETalbqGJKXtr6rllscX88bGPfzwk8P51JiTw44kknFUCCSlffW3b/Lmpg944IaR/MNZfcOOI5KRVAgkpTSeSrpvUQHXjerHFy8cyPghJ4YdTSRj6RqBpIymU0mX763kly+tZ9+h2rCjiWQ0FQJJCTWRen7wzKqjppKurK3TVNIiAVMhkNC99u5uzr33RfYcjP/OX1NJiwRLhUA63L7KWn67aCMLynYAMLi4kPMG9aJXt/y422sqaZFg6WKxtLumF3ynThjCJ87qy2vrdzNr8Waee+t9qiP1XD+qH+OHnEjvws489OmzD18jaNw9pKmkRYKnQiDtqunBvHxvJXfPXskjL7/L29sOcFyXPK4f1Z/rR/VnWMlxR7RtmDm0aRHRjKIiwVIhkHb1o+fWxL3g+/7+ah64cSSXDy2mS6fEM4VeM7JEB36RDqZCkKHidc+05ADb0L58byUli+Yf0f5QTYT3dh1iw66DjDi5iJKiAl5eu5PbfreMPQdr4r7eBwdr+IRuCBNJSYEWAjObCPwMyAUedfdpTdZbbP2VwCHg8+7+ZnvnONZBrSXtW3NQDaNtou4ZoNn27s6TizfzvadWURWpP9z+zj+s4Ofz36Giuo7391cd3n76pDO5blR/SnoWMOGMYv68Yhv7qyJHva4u+IqkrsAKgZnlAg8BlwFbgDfM7Gl3f7vRZlcAg2OPMcDDsX/bTVsOim1t395t75q9gorqCJcPLaamrp5InVOQn0vxcV0AWL55L9WRev79z2/H7Z757lNvsWrrPiqqIxyoilBRHeGyocV8eswp7DtUy4X3zaeiOkK9H52nOlLPxj2H+MRZJQzs3ZWBvQsZ0Lsrg3oXAvCREwq599ozGTOwly74iqSZIM8IRgPr3H09gJnNBK4GGheCq4HfuLsDi8ysyMz6uPu29goxfV5Z3IPiHU8u58H579ApN4fnbrsIgHufXc0Lb2+PbhQ7GG7+4BC1dX5U+9ufXM79L5TRp0cBs758HgDfmLmUxe99EG3uzvv7q446qFbW1vHPs5bxr0+vwt0ZcXJPvjAouu7jD7zCO9srqHcnEudoXFVbz3fmvMV35rx1eNnHh/fhoU+fDcBnfvV3DsR5N97gQFWE//37Jrp1zqN75zwKu+RRE3vX37VzLtee3Y/Cznn8fMG6uO0jdc7915+V8PVBF3xF0pFFj8EBvLDZJGCiu98Se34TMMbdpzTa5hlgmru/Gnv+InCnuy9u8lqTgckAxcXFpTNnzkw6x+efO5hw3eiTcsk1+PJZ0XfUf9lYy9oPPiwaBvz9/boEreG8vrkcl2/ceHpnAOaur6G8wrHYLMmvlic+KF9ych45BicU5DC2dzWFhYU8t6GWAzXR9s+sTzytwmeH5pOXA7kGJ3TN4bSe0Yuvb++uwx1+uaKK/XG66nt1Me4f1zXh6za4feEhdlcd/XeRbPuOUFFRQWFhYdgxjqJcLaNcLdOWXOPHj1/i7qPirQvyjCDepPFNjy7JbIO7PwI8AjBq1CgfN25c0iFKFs2nPM6dqSVFBcy67eIjlsV71fOnJW7/xNebtG/yAsdq+6uvfth24cKFjBs37oj2S4/R9gefvfio5Y3zl3wk/nj87149nHFJvDP/bo+2te8IDfss1ShXyyhXywSVK8g7i7cA/Rs97wdsbcU2bTJ1whAKmgxXbEmfdVvah9X2mpEl3HvtcEqKCjCixePea4cn3T3TuD2taC8i6SXIM4I3gMFmNhAoB24APtVkm6eBKbHrB2OAfe15fQCO7LMu31tJSQv7rNvS5x1W24b2bTlwN7RP1XdGItJ+AisE7h4xsynAPKLDRx9z91Vmdmts/QxgLtGho+uIDh+9OYgsbT2oteWgGlZbEZFkBXofgbvPJXqwb7xsRqOvHfhakBlEROTYNPuoiEiWUyEQEclyKgQiIllOhUBEJMsFdmdxUMxsJ7Cxlc17A7vaMU57SdVckLrZlKtllKtlMjHXKe5+QrwVaVcI2sLMFie6xTpMqZoLUjebcrWMcrVMtuVS15CISJZTIRARyXLZVggeCTtAAqmaC1I3m3K1jHK1TFblyqprBCIicrRsOyMQEZEmVAhERLJcxhUCM7vOzFaZWb2ZjWqy7m4zW2dmZWY2IUH7483sBTN7J/ZvzwAy/s7MlsUe75nZsgTbvWdmK2PbLY63TTvn+jczK2+U7coE202M7cN1ZnZXB+SabmZrzGyFmf3RzIoSbNch+6u5n9+iHoitX2FmZweVpdH37G9mC8xsdezv/xtxthlnZvsa/X6/F3SuRt/7mL+bkPbZkEb7YpmZ7Tez25ps0yH7zMweM7MdZvZWo2VJHYva5f+ju2fUA/goMARYCIxqtHwosBzoDAwE3gVy47S/D7gr9vVdwI8Czns/8L0E694Denfgvvs34I5mtsmN7btBQH5snw4NONflQF7s6x8l+p10xP5K5ucnOrX6s0Q/ge9c4O8d8LvrA5wd+7o7sDZOrnHAMx3199SS300Y+yzO7/V9ojdddfg+Ay4CzgbearSs2WNRe/1/zLgzAndf7e5lcVZdDcx092p330D0MxBGJ9ju17Gvfw1cE0hQou+CgOuBJ4L6HgEYDaxz9/XuXgPMJLrPAuPuz7t7wwdALyL6SXZhSebnvxr4jUctAorMrE+Qodx9m7u/Gfv6ALAaSKcPs+jwfdbEJcC77t7aWQvaxN1fBvY0WZzMsahd/j9mXCE4hhJgc6PnW4j/H6XYY5+SFvv3xAAzXQhsd/d3Eqx34HkzW2JmkwPM0diU2Kn5YwlORZPdj0H5AtF3jvF0xP5K5ucPdR+Z2QBgJPD3OKvPM7PlZvasmZ3RUZlo/ncT9t/VDSR+QxbWPkvmWNQu+y3QD6YJipn9BTgpzqp/cfenEjWLsyywsbNJZryRY58NnO/uW83sROAFM1sTe+cQSC7gYeAeovvlHqLdVl9o+hJx2rZ5Pyazv8zsX4AI8L8JXqbd91e8qHGWNf35O/Rv7YhvbFYI/AG4zd33N1n9JtGuj4rY9Z85wOCOyEXzv5sw91k+8Ang7jirw9xnyWiX/ZaWhcDdL21Fsy1A/0bP+wFb42y33cz6uPu22KnpjiAymlkecC1QeozX2Br7d4eZ/ZHoaWCbDmzJ7jsz+y/gmTirkt2P7ZrLzD4HXAVc4rHO0Tiv0e77K45kfv5A9lFzzKwT0SLwv+4+u+n6xoXB3eea2S/MrLe7Bz65WhK/m1D2WcwVwJvuvr3pijD3Gckdi9plv2VT19DTwA1m1tnMBhKt6q8n2O5zsa8/ByQ6w2irS4E17r4l3koz62Zm3Ru+JnrB9K1427aXJn2yn0zw/d4ABpvZwNg7qRuI7rMgc00E7gQ+4e6HEmzTUfsrmZ//aeCzsZEw5wL7Gk7xgxK73vQrYLW7/zjBNifFtsPMRhP9/787yFyx75XM76bD91kjCc/Mw9pnMckci9rn/2PQV8M7+kH0ALYFqAa2A/MarfsXolfYy4ArGi1/lNgII6AX8CLwTuzf4wPK+Thwa5NlfYG5sa8HER0BsBxYRbSLJOh99z/ASmBF7I+pT9NcsedXEh2V8m4H5VpHtB90WewxI8z9Fe/nB25t+H0SPV1/KLZ+JY1GrwWY6QKiXQIrGu2nK5vkmhLbN8uJXnQfG3SuY/1uwt5nse/bleiBvUejZR2+z4gWom1Abez49cVEx6Ig/j9qigkRkSyXTV1DIiIShwqBiEiWUyEQEclyKgQiIllOhUBEJMupEIi0gUVn/dxgZsfHnveMPT8l7GwiyVIhEGkDd99MdGqOabFF04BHPKTJy0RaQ/cRiLRRbGqHJcBjwJeAkR6dCVIkLaTlXEMiqcTda81sKvAccLmKgKQbdQ2JtI8riE4RMCzsICItpUIg0kZmNgK4jOgna32zgz9QRaTNVAhE2iA2M+XDROf/3wRMB/4z3FQiLaNCINI2XwI2ufsLsee/AE43s4+FmEmkRTRqSEQky+mMQEQky6kQiIhkORUCEZEsp0IgIpLlVAhERLKcCoGISJZTIRARyXL/H2W6VY975DGjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = 1 / (1 + np.exp(-x))\n",
    "plot_graph(y, \"Sigmoid(X)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b54decc",
   "metadata": {},
   "source": [
    "### Tanh\n",
    "\n",
    "The hyperbolic tangent function is also a prevalent activation function for neural networks that must output values in the range between __-1 and 1__. This activation function is simply the hyperbolic tangent (tanh) function, as shown in the following equation:\n",
    "\n",
    "$$ \n",
    "\\begin{split}\n",
    "            tanh(x) & = \\frac{e^x - e^{-x}}{e^x + e^{-x}} \\\\\n",
    "                    & = \\frac{e^{2x} - 1}{e^{2x} + 1}\n",
    "\\end{split}                  \n",
    "$$\n",
    "\n",
    "\n",
    "Like the sigmoid neuron, its activations saturate, but unlike the sigmoid neuron its output is zero-centered. Therefore, in practice the tanh non-linearity is always preferred to the sigmoid nonlinearity. Also note that the tanh neuron is simply a scaled sigmoid neuron, in particular the following holds: \n",
    "\n",
    "$$ tanh(x) = 2 \\sigma \\left( 2x \\right) -1 $$\n",
    "\n",
    "As for Sigmoid, Tanh is highly compute-intensive and suffers from saturation problem and thus vanishing gradient. In fact, when the neuron reaches the minimum or maximum value of its range, that respectively correspond to -1 and 1, its derivative is equal to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5d5efdbd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEGCAYAAABLgMOSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqYklEQVR4nO3de3xU9Z3/8dcnIUAgQAQhQFDxgqhcvJCirdsavIFuW9StXe3Wte1afralj9ZaV/x17c/W7Upra3e3tVXautptbdpuhVpLpV7IWrVWQS4BEUHwkgmCXBIICZDL5/fHnNAhzCQzk5k5k+T9fDzmkTnnfL8z75wk88n5npu5OyIiIqkqCDuAiIj0TiogIiKSFhUQERFJiwqIiIikRQVERETSMiDsALl07LHH+sSJE9Pqu3//foYOHZrZQBmgXKlRrtQoV2ryNRf0LNvKlSt3uvvooxa4e795zJgxw9O1fPnytPtmk3KlRrlSo1ypyddc7j3LBqzwOJ+pGsISEZG0qICIiEhaVEBERCQtKiAiIpIWFRAREUlLqIfxmtkDwAeBHe4+Nc5yA/4DuBxoAj7h7i8Hy+YEywqBH7v7wpwFF5G0LVkV4e5lG6mrb2Z8aTG3zJ7MFWeXp9Q3Ut9M+QtPp9Q3U+/dm/rG9k93nXUl7PNAHgS+D/w0wfLLgEnB41zgh8C5ZlYI3AtcAtQCL5nZo+7+StYTi0jaH2pLVkW47ZEamlvaAIjUN3PbIzUA3fbvSd8w37u3fs/JCLWAuPszZjaxiyZzgZ8GxyG/YGalZjYOmAhsdvctAGZWFbRVARHJsngfSgt+s5Y9TYe44NTRtLU7k8qGAfDqO3vZ1nCAQ63tHGpt52u/W3+4X4fmlja++tt1vLP3AO5QOqSIa2ceD8DiVbVE9jTjDov+tCVu39t/u46tO/cfnjfhmGKurjgOgP96bit7mloOP0/Uf0tM/9PGDuPyaeMA+N5Tm2hpa+e/nnsjft8l69jybuPheW+8eQjG7aBy8hgOtrZx79ObE/b95uOvHtG3wwdOHU3FxJHs3n+I25esi9v37mUbee/Jo/j5C28e1f+yaeM4fdxw3t7d1GX/TBQQ85DvBxIUkMcSDGE9Bix092eD6aeAW4kWkDnufkMw/zrgXHefH+c15gHzAMrKymZUVVWllbOxsZGSkpK0+maTcqVGuVLTkcvd2dnsbN3bzn+tO0hza+I+gwvhvkuiZzz/cPUB/vJOW+LGcYwdYiz8wBAA7vpLMxv3tKfUf/IxBdx2bjEAtz7TxPam1D7jzhtXyI1nDgbgxif2c7ANunoFO2LKmT2xiGtOG0Rzq/PZJ5tS6Bv10ckDuezEIrY1tnPbs80J+97x3sF87c8Hjpr/f6YP4rzxA9iwq41vvnT08g4Pzkn+rPRZs2atdPeKzvPDHsLqTrz1613MP3qm+yJgEUBFRYVXVlamFaS6upp0+2aTcqVGuf4q0TBUxz+VZsZ//vpJXtpeTE2kgfrgP/mufPfvz6S4qJDKqdH/4I+b0khDcwsDCwsYNKCAj//kL2zfe/CofuNGDOapmy/AMMxgcFEhAOe/vx13MIML7l5OXf3RH4jlpcU8t+DCuHn+UvnX5+cvfJpI/dEfyF31f7Uytb6df45bL07vfTt8f13ivp+YeyGfmJu4byXws02J+2fi9y3fj8KqBY6LmZ4A1HUxX0SS0DEMFalvxokOQ33512uY/e/PcM6dT7CmtgGAg22we/8h5kwZyzeunMqj889n/IjBcV+zvLSYK8+ewJygeACcPLqEc44/hqnlI5hUNozbLjud4qA4dCguKuTWOacxZOAAigcWHi4eAEWFBQwcUEBRYQH/PPu0uH1vmT05qe/5ltmT0+7fG/tmon938n0L5FFgfrCP41ygwd23mdm7wCQzOxGIANcAHwsxp0ivcveyjUeNjbe2O6/vaOSqc8oPf+i8Z+wAbrnm/Ue0++c5px2xDwSS/1DqGHdPZwd8bN9IfTPlKR6RlKn37i19O/dPZ511J+zDeH9BdEvrWDOrBf4fUATg7vcBS4kewruZ6GG8nwyWtZrZfGAZ0cN4H3D39Tn/BkR6qbo4wxoAbe3Otz5yZpd9M/Ghlu4HWEffdIf8MvHevalvbP9sDJOGfRTWtd0sd+BzCZYtJVpgRCRFI4qLqG8+ep/G+NLipPr39ENN+oZ83wciIhn20hu7aWhuobDToSiZHBuX/kEFRKQf2dl4kPkPv8wJo4bwr1dOpby0GCO6A/yuq6Zpq0JSku870UUkQ9ranS9WrWZPUwsPfPY9TBk/gmtnnhB2LOnFtAUi0k8sXhXh2c07+dqHpzBl/Iiw40gfoC0QkX7iyrPLKRlUyOwpY8OOIn2ECohIH7d9b/Ts7bLhg484yU+kpzSEJdKHtbS1M//hl7n6vj/T0pbaNaVEuqMCItKHfXvZRl56Yw83X3oqRYX6c5fM0m+USB/1xCvbuf+ZLfzDuccz9ywdniuZpwIi0ge9vbuJm3+1mqnlw7n9g2eEHUf6KO1EF+mDhhcXcfEZZXzxolOPuLqtSCapgIj0Ma1t7YwoLuKej54VdhTp4zSEJdKHPLqmjit/8Dzv7jv6pk0imaYCItJHvP5uI7f9Zi2DBhRQOqQo7DjSD6iAiPQBzYfa+OzPXmZQUSHf+9jZOmRXckL7QER6sY77mnfc9/rGC05i3Ijk7ukh0lOh/ptiZnPMbKOZbTazBXGW32Jmq4PHOjNrM7ORwbI3zKwmWLYi9+lFwhV7X/MODz3/JktWRUJMJf1JaAXEzAqBe4HLgDOAa83siAPW3f1udz/L3c8CbgP+1913xzSZFSyvyFVukXwR777mzS1t3L1sY0iJpL8JcwtkJrDZ3be4+yGgCpjbRftrgV/kJJlIL5DovuaJ5otkmkVvOx7CG5t9BJjj7jcE09cB57r7/DhthwC1wCkdWyBmthXYAzhwv7svSvA+84B5AGVlZTOqqqrSytvY2EhJSUlafbNJuVLTl3LdXN3ErgNH//2OGmx8p3JIaLlyQblS15Nss2bNWhlvpCfMnegWZ16iavYh4LlOw1fnu3udmY0BnjCzV939maNeMFpYFgFUVFR4ZWVlWmGrq6tJt282KVdq+lKu20dE+NKvVtMe81dTXFTI7XOnUZmhW9P2pfWVC/maC7KTLcwhrFrguJjpCUBdgrbX0Gn4yt3rgq87gMVEh8RE+o0PnzmeogJj6MBC3ddcQhHmFshLwCQzOxGIEC0SH+vcyMxGABcAH4+ZNxQocPd9wfNLga/nJLVInnhj134Otjl3XjGFj77nuO47iGRYaAXE3VvNbD6wDCgEHnD39WZ2Y7D8vqDplcAf3X1/TPcyYLGZQfR7eNjdH89depHw1UQaAJg2Qfc3l3CEeiKhuy8Flnaad1+n6QeBBzvN2wKcmeV4InntvJNGcc9Hz2TSmPzcaSt9n85EF+mlyoYP5qpzJoQdQ/oxXTBHpBdqa3eqXnyL2j1NYUeRfkwFRKQX2rqzkQWP1PDClt3dNxbJEhUQkV7o8A70cu1Al/CogIj0QmtrGyguKuTk0UPDjiL9mAqISC9UU9vAlPHDGaD7fkiI9Nsn0su0tTvr6/YyVcNXEjIdxivSyxQWGM8vuJCWtvawo0g/pwIi0gsdM3Rg2BFENIQl0ttUvfgWP/7TlrBjiKiAiPQ2v1zxNn98ZXvYMURUQER6k9a2dl6p28t07UCXPKACItKLbNrRyMHWdl2BV/KCCohIL1JTqzPQJX+ogIj0IrubDjF62CAmjtIZ6BI+HcYr0ovceMHJzHv/SRQUWNhRRMLdAjGzOWa20cw2m9mCOMsrzazBzFYHj68m21ekr1LxkHwRWgExs0LgXuAy4AzgWjM7I07TP7n7WcHj6yn2FekzNmzby5U/eI51wZV4RcIW5hbITGCzu29x90NAFTA3B31FeqXVb9ez6q16hg3WyLPkB3P3cN7Y7CPAHHe/IZi+DjjX3efHtKkEfgPUAnXAl919fTJ9Y15jHjAPoKysbEZVVVVaeRsbGykpyb97TytXanpzrgfXHeTFd1q596IhmOVmGKs3r68w5Gsu6Fm2WbNmrXT3is7zw/xXJt5fQOdq9jJwgrs3mtnlwBJgUpJ9ozPdFwGLACoqKryysjKtsNXV1aTbN5uUKzW9Odd3ap7l7IkDmDXrvNyEonevrzDkay7ITrYwh7BqgeNipicQ3co4zN33untj8HwpUGRmxybTV6QvOdjaxqvv7GVaeWnYUUQOC7OAvARMMrMTzWwgcA3waGwDMxtrwba6mc0kmndXMn1F+pK9za3MmjyGc08aGXYUkcNCG8Jy91Yzmw8sAwqBB4L9GzcGy+8DPgJ8xsxagWbgGo/utInbN5RvRCQHRg8bxKJ/PGoIWiRUoR7OEQxLLe00776Y598Hvp9sX5G+6kBLG4OLCsOOIXIEXcpEpBe46gfP86Vfrg47hsgRVEBE8tyBljZe276PcaWDw44icgQVEJE8t2HbXlrbXUdgSd5RARHJczXBpUt0DxDJNyogInmupraBUUMHMn6EhrAkv+iiOiJ57uIzypgyfnjOLl8ikiwVEJE8N3vK2LAjiMSlISyRPLaz8SCbd+yjvT2ci56KdEUFRCSPPbamjovveYYd+w6GHUXkKCogInlsbaSB0cMGUTZ8UNhRRI6iAiKSx9ZFGphePkI70CUvqYCI5Kn9B1vZvKNR539I3lIBEclTr2zbS7vDdBUQyVMqICJ5avLYYfzk+gpmnKB7gEh+0nkgInlq+OAiLjq9LOwYIgmFugViZnPMbKOZbTazBXGW/4OZrQ0ez5vZmTHL3jCzGjNbbWYrcptcJPse/stbvFK3N+wYIgmFVkDMrBC4F7gMOAO41szO6NRsK3CBu08H7gQWdVo+y93Pcnfdqk36lH0HWvi/i2t4csP2sKOIJBTmFshMYLO7b3H3Q0AVMDe2gbs/7+57gskXgAk5zigSivXBloeOwJJ8ZtFbjIfwxmYfAea4+w3B9HXAue4+P0H7LwOnxbTfCuwBHLjf3TtvnXT0mwfMAygrK5tRVVWVVt7GxkZKSkrS6ptNypWa3pLrD1tb+OXGQ/znrCEMHxTeOSC9ZX3li3zNBT3LNmvWrJVxR3rcPZQHcDXw45jp64DvJWg7C9gAjIqZNz74OgZYA3ygu/ecMWOGp2v58uVp980m5UpNb8n1+Ydf9vfd9VQ4YWL0lvWVL/I1l3vPsgErPM5naphDWLXAcTHTE4C6zo3MbDrwY2Cuu+/qmO/udcHXHcBiokNiIn3CK9v2MrV8eNgxRLoU5mG8LwGTzOxEIAJcA3wstoGZHQ88Alzn7q/FzB8KFLj7vuD5pcDXc5ZcJMse+/zfsPdAS9gxRLoUWgFx91Yzmw8sAwqBB9x9vZndGCy/D/gqMAr4QXAtoFaPjsOVAYuDeQOAh9398RC+DZGsGFxUyOCiwrBjiHQp1BMJ3X0psLTTvPtint8A3BCn3xbgzM7zRfqC362pY21tPQsuO53CAl1EUfKXLmUikmceX/8Oj69/R8VD8l7SWyBmdgwwHmgG3nD39qylEunHamobmFau8z8k/3VZQMxsBPA54FpgIPAuMBgoM7MXgB+4+/KspxTpJ+qbDvHW7iaunXl82FFEutXdFsj/AD8F3u/u9bELzGwGcJ2ZneTuP8lSPpF+pSbSAKAtEOkVuiwg7n5JF8tWAisznkikH6tvauHYkoEqINIrdLkT3cz+LsH8gWZ2e3YiifRfHzpzPC995WJGDCkKO4pIt7o7Cmuemf0hONkPADO7DFhL9PwMEckw3f9ceovuhrBmm9m1wJNm9jAwFRgN/L27r8lFQJH+Yvf+Q3zkvuf5l789nQtP042kJP8lcxjvr4ApwE1APXBh7GVFRCQzaiINbHl3v85Al16ju30gfwOsIjpcdRwwH/idmX3dzAblIJ9Iv1FTWw/AVO1Al16iu30g/w7c4O6fcfc97r4EOBsYRPQS6iKSIWtrGzjp2KEMH6wd6NI7dDeENbPzGefu3gTcamYPZi2VSD9UE2ngPRNHhh1DJGndbYG8L9ECd99gZsPNbGqGM4n0O63tzvmnHMus00aHHUUkad1tgfydmX0LeJzoSYMdlzI5hehdAk8Abs5qQpF+YECB8e2rdYFp6V26O4z3puAiih8hegvacUQvpriB6H3In81+RJG+r7k1eotQnQMivUm3h/G6+x7gR8FDRLLg/jUHuX/j8yz53PlhRxFJWndX4/1SV8vd/Z6evLmZzQH+g+gdCX/s7gs7Lbdg+eVAE/AJd385mb4ivcGSVRHuXraRSH0bxUX7WLIqwhVnl4cdSyQp3e1EHxbz+HKn6WE9eWMzKwTuBS4DzgCuNbMzOjW7DJgUPOYBP0yhr0heW7Iqwm2P1BCpbwaguaWN2x6pYcmqSMjJRJLT3T6Qr3U8N7MrYqczYCawObg9LWZWBcwFXolpMxf4qbs78IKZlZrZOGBiEn1F8trdyzbS3NJ2xLzmljbuXrZRWyHSK6RyT3TP8HuXA2/HTNcC5ybRpjzJvgCY2TyiWy+UlZVRXV2dVtjGxsa0+2aTcqUmn3J1bHnEm58vGfNpfcVSrtRlI1sqBSTT4h1u0rlIJWqTTN/oTPdFwCKAiooKr6ysTCHiX1VXV5Nu32xSrtTkU67yF56OW0TKS4vzJmM+ra9YypW6bGTr7lpYNWa21szWAqd1PO+Y38P3riV6fa0OE4C6JNsk01ckr90yezLFnS6cWFxUyC2zJ4eUSCQ13W2BfDCL7/0SMCm410gEuAb4WKc2jwLzg30c5wIN7r7NzN5Noq9IXrvi7HKaWlq554+vsbPxEOWlxdwye7L2f0iv0d1O9Dez9cbu3mpm84FlRA/FfcDd15vZjcHy+4ClRA/h3Uz0MN5PdtU3W1lFsmXU0EHsbDzEv5w3mBuuuDDsOCIpSWofiJldBXwTGEN0/4MB7u7De/Lm7r6UaJGInXdfzHMHPpdsX5HeZl2kgcIC4/hh3R1RL5J/kt2J/i3gQ+6+IZthRPqbtbUNTBpTwsDC9u4bi+SZZP/t2a7iIZJZ7k5NpIHpE3QDKemdkt0CWWFmvwSWAAc7Zrr7I9kIJdIf1DUcYPf+Q0ybUAoH9oQdRyRlyRaQ4UR3Yl8aM88BFRCRNB1bMpBf3/hejh85hA0vbw07jkjKkiog7v7JbAcR6W8GDSg8fAdCjQ9Lb5TsUViDgX8CphC9oRQA7v6pLOUS6fN+teJtJhxTzPtOPjbsKCJpSXYn+n8DY4HZwP8SPfN7X7ZCifR17s6/Ld3A79boAgrSe3V3KZOOLZRT3P12YL+7PwT8LTAt2+FE+qraPc3UN7UwtVxHYEnv1d0WyIvB15bga72ZTQVGEL2kuoikYW1tAwDTy0vDDSLSA8kehbUouDf6vxC9PlUJcHvWUon0cWsj9QwsLODUsSVhRxFJW3cFZEzMbW07jsS6N/g6NDuRRPq+zdsbmTx2GIMGFHbfWCRPdVdAColubSR9/w0R6d6P/rGC+uaW7huK5LHuCsg2d/96TpKI9CMFBcbIoQPDjiHSI93tRI+35SEiPfC/r73Lrf+zlvqmQ2FHEemR7grIRTlJIdKPPLvpXRavjjB0UJh3lBbpuS4LiLvvzlUQkf5ibW0DZ4wbTlGh7gEivVsov8FmNtLMnjCzTcHXY+K0Oc7MlpvZBjNbb2ZfiFl2h5lFzGx18Lg8t9+BSHra2531dXuZphMIpQ8I61+gBcBT7j4JeCqY7qwVuNndTwfOAz5nZmfELP+uu58VPHRnQukVtu7aT+PBVqbpHiDSB4RVQOYCDwXPHwKu6NzA3be5+8vB831EL1hanquAItmwZ/8hTjx2qG4iJX2CRW87nuM3Nat399KY6T3uftQwVszyicAzwFR332tmdwCfAPYCK4huqcS9I4+ZzQPmAZSVlc2oqqpKK3NjYyMlJfl31rBypUa5UqNcqcnXXNCzbLNmzVrp7hVHLXD3rDyAJ4F1cR5zgfpObfd08TolwErgqph5ZURPciwAvgE8kEymGTNmeLqWL1+edt9sUq7UKFdqlCs1+ZrLvWfZgBUe5zM1a8cRuvvFiZaZ2XYzG+fu28xsHLAjQbsi4DfAzz3m9rnuvj2mzY+AxzKXXCQ72tqdC79TzafffxIfP++EsOOI9FhY+0AeBa4Pnl8P/LZzAzMz4CfABne/p9OycTGTVxLdshHJa1vebeTNXU0UF+n6V9I3hFVAFgKXmNkm4JJgGjMbb2YdR1SdD1wHXBjncN1vmVmNma0FZgE35Ti/SMo6LuGuI7CkrwjlVFh330Wcs9zdvQ64PHj+LAkupeLu12U1oEgW1EQaKC4q5OTR+bmTVSRVOhVWJEdqIg1MLR9OYYEuMSd9gy7GI5Ij5500kjHDBocdQyRjVEBEcuSW2aeFHUEkozSEJZIDDU0tHGptDzuGSEapgIjkwHee2Mh5dz3VcSKsSJ+gAiKSA2trG5g0poTo6U0ifYMKiEiWtbS188q2vbqAovQ5KiAiWbZpeyOHWtuZqnuASB+jAiKSZTWRegCmTygNNYdIpqmAiGTZOccfw22XncYJI4eEHUUko3QeiEiWTSobxqSyYWHHEMk4bYGIZNGh1naWv7qDhqaWsKOIZJwKiEgWvbZ9H5988CX+tPndsKOIZJwKiEgWdVzCfXp5abhBRLJABUQki2oiDYwoLuK4kcVhRxHJuFAKiJmNNLMnzGxT8PWYBO3eCG4ctdrMVqTaXyRsNZF6pk8YoTPQpU8KawtkAfCUu08CngqmE5nl7me5e0Wa/UVCcaCljY3v7NMJhNJnhXUY71ygMnj+EFAN3JrD/iJZN7CwgMc+/36GDNQ90KVvsjCuDmpm9e5eGjO9x92PGoYys63AHsCB+919USr9g2XzgHkAZWVlM6qqqtLK3NjYSElJ/t2KVLlSo1ypUa7U5Gsu6Fm2WbNmrew0ChTl7ll5AE8C6+I85gL1ndruSfAa44OvY4A1wAeC6aT6d37MmDHD07V8+fK0+2aTcqUml7kWv1zrv1sTSaqt1ldqlCt1PckGrPA4n6lZG8Jy94sTLTOz7WY2zt23mdk4YEeC16gLvu4ws8XATOAZIKn+ImG6/5ktjB42iA9OHx92FJGsCGsn+qPA9cHz64Hfdm5gZkPNbFjHc+BSolswSfUXCdOBljZe276P6dqBLn1YWAVkIXCJmW0CLgmmMbPxZrY0aFMGPGtma4AXgd+7++Nd9RfJF69s20tbuzNN9wCRPiyUo7DcfRdwUZz5dcDlwfMtwJmp9BfJF+siwRnoKiDSh+lMdJEseHNXE8eWDGTs8MFhRxHJGl3OXSQLbv/gGdx0yak6A136NG2BiGRJySD9fyZ9mwqISIbV1DbwmZ+t5I2d+8OOIpJVKiAiGbbizd38Yd07FOsSJtLHqYCIZFhNbQNjhg2iTDvQpY9TARHJsLWRBh2+K/2CCohIBjUebOX1dxt1CXfpF1RARDJoV+NBpk8o5ezjdY8z6ft0nKFIBp0waii//dz5YccQyQltgYhkkIdwfx2RsKiAiGTQh77/LN98/NWwY4jkhAqISIbsO9DCushehhTp/A/pH1RARDJkXWQvgC7hLv2GCohIhtRE6gGYpkN4pZ9QARHJkLW1DZSXFjOqZFDYUURyIpQCYmYjzewJM9sUfD3qoHkzm2xmq2Mee83si8GyO8wsErPs8px/EyKdnHvSKP7hvOPDjiGSM2GdB7IAeMrdF5rZgmD61tgG7r4ROAvAzAqBCLA4psl33f3buYkrktiSVRHuXraRuvpmxpcWM35EMVecXR52LJGsC2sIay7wUPD8IeCKbtpfBLzu7m9mM5RIqpasinDbIzVE6ptxIFLfzG2P1LBkVSTsaCJZZ2Gc+GRm9e5eGjO9x90TXvvBzB4AXnb37wfTdwCfAPYCK4Cb3X1Pgr7zgHkAZWVlM6qqqtLK3NjYSElJSVp9s0m5UpPpXDdXN7HrwNF/Q6MGG9+pHBJarkxRrtTkay7oWbZZs2atdPeKzvOzVkDM7ElgbJxFXwEeSraAmNlAoA6Y4u7bg3llwE7AgTuBce7+qe4yVVRU+IoVK1L9VgCorq6msrIyrb7ZpFypyXSuExf8nnh/QQZsXfi3Sb9Of1lfmaJcqetJNjOLW0Cytg/E3S/uIsx2Mxvn7tvMbBywo4uXuozo1sf2mNc+/NzMfgQ8lonMIqkaXzqYSP2BOPOLQ0gjklth7QN5FLg+eH498Nsu2l4L/CJ2RlB0OlwJrMtoOpEkvWfiyKPmFRcVcsvsySGkEcmtsArIQuASM9sEXBJMY2bjzWxpRyMzGxIsf6RT/2+ZWY2ZrQVmATflJrbIkU4ZU8KZE4ZTXjoYA8pLi7nrqmk6Ckv6hVAO43X3XUSPrOo8vw64PGa6CRgVp911WQ0okqT5F07is5WnUFBgYUcRyTmdiS6SorZ256Zfrua5zTsBVDyk31IBEUnRvcs3s3hVhDd3NYUdRSRUKiAiKXhu806+++RrXHl2OdfOPC7sOCKhUgERSdL2vQf4QtUqTh5dwr9eMRUzDV1J/6Z7oosk6RcvvsX+g2384tPnMHSQ/nRE9FcgkqQvXDSJD04fxyljhoUdRSQvaAhLpBvPv76Tt3Y1YWYqHiIxVEBEuvD27iY+87OXufU3a8OOIpJ3VEBEEjjY2sbnHn6ZdncW/t20sOOI5B3tAxFJ4N9+v4G1tQ3c9/EZnDBqaNhxRPKOtkBE4njyle089Oc3+ae/OZE5U+PdlUBEtAUiEsf7ThnFzZecyo2VJ4cdRSRvqYCIcOR9zceOGMytc07j8xdNCjuWSF5TAZF+r+O+5s0tbQBsazjAgkeiR13psuwiiWkfiPR7dy979XDx6HCgpZ27l20MKZFI76AtEOkzYoehxpcWc8vsyUdtQbg7dfXN7Go8xLQJIwDi3pIWoK6+OeuZRXqzUAqImV0N3AGcDsx09xUJ2s0B/gMoBH7s7h13LhwJ/BKYCLwBfNTd92Qja8eHUqS+mfIXno77odRd364+0LLVP+y+uV5fnYehIvXN3PZIDQBjhg/iz6/voibSwMqtTexb9jSnlpXwx5suAGD44AHsPdB61GvqvuYiXQtrC2QdcBVwf6IGZlYI3Ev0lra1wEtm9qi7vwIsAJ5y94VmtiCYvjXTIbv6UOrug60nfcN873zs+6Ezx3OwtY1Dre0cam3nYPA4YdQQigoLeHt3E3c+9spRw1DNLW3cvWwj5544kiWrI5xaNowzRw/g0orJTJ9Qerjd1+dOPeK9Qfc1F0lGWLe03QB0dznsmcBmd98StK0C5gKvBF8rg3YPAdVkoYDcvWxj3A+lL/96Dfcu30zFxGO466rpAHzsRy/w7r6Dh9tt3bmf1nZP2Bdg9pSxfDn4kJrz78/QFtM+mf7Xzjyek4D6pkNcfd+fk+r7/aAvwOcvPIW5Z5WzeUcjN/5sJQBvdNH3e09vOmL+HR+ewvsnjebFrbtZ8Mha3trVFLfvnY+9wj1PvIbjeLDYHe6/bgZTy0fw6Jo6vvSr1XTqergA7Gw8yL/+fgOdPb/gQsaXFrN4VYRd+w8dtRyiw1C3XX4637hyGsUDC6murqbyvROPaNNR3HqytSjSH+XzPpBy4O2Y6Vrg3OB5mbtvA3D3bWY2JtGLmNk8YB5AWVkZ1dXVSQeIJBgDb213RlgzB/ccpLp6NwCDDh1khPkRbbrqC9Cw/S2qq7cBMJwDtNuR7brrv+Ot1xkz/ADPP/ccI+xgUn2PKfjr9/Tmpg1U12/i3aZ2jimIfgBv7qLvqMIj9xVsWr+Wtkghb+5t49jCFrYk6Ltr/yFOHd4GBkb0mzRg/eqV7NxUwDt72o4qHh0i9c0U7t7KR08tYkCBMaAAigqgqMBYu+IFXhtgjD/UzvCBsDdODRk52Fi/8q/FtbGxMe7vQCnwjfMKgOCM84ZNVFdvOqpdtiTKFTblSk2+5oLsZMtaATGzJ4F4p/B+xd1/m8xLxJmX4GMmMXdfBCwCqKio8MrKyqT7lr/wdNwiUl5azP/cdOER8zq/7PkLk+/bk/7V1dVUVlZy+SXJ9f31F49+b4CrL+++768S9AW4vpu+v/hC4r6VwEMbE/f95NzEfTsMGBuJOwx1+9xpVMZsSXSsr3yjXKlRrtRlI1vWDuN194vdfWqcRzLFA6JbHLH3DJ0A1AXPt5vZOIDg647MJf+rW2ZPprio8Ih5yY6N96RvmO/dG/tCdBjqrqumUV5ajBEtPHddNU3DUCJZlM9DWC8Bk8zsRCACXAN8LFj2KNF/ehcGX5MtSimJHRuP1DdTnsLYeE/H1XvSPx/65np9dbyGCoZIDrl7zh/AlUS3MA4C24FlwfzxwNKYdpcDrwGvEx366pg/CngK2BR8HZnM+86YMcPTtXz58rT7ZpNypUa5UqNcqcnXXO49ywas8DifqWEdhbUYWBxnfh3RotExvRRYGqfdLuCibGYUEZGu6VImIiKSFhUQERFJiwqIiIikRQVERETSYu4pn5vXa5nZu8CbaXY/FtiZwTiZolypUa7UKFdq8jUX9CzbCe4+uvPMflVAesLMVrh7Rdg5OlOu1ChXapQrNfmaC7KTTUNYIiKSFhUQERFJiwpI8haFHSAB5UqNcqVGuVKTr7kgC9m0D0RERNKiLRAREUmLCoiIiKRFBSSGmV1tZuvNrN3MKjotu83MNpvZRjObnaD/SDN7wsw2BV+PyULGX5rZ6uDxhpmtTtDuDTOrCdqtyHSOOO93h5lFYrJdnqDdnGAdbg7uZ5/tXHeb2atmttbMFptZaYJ2OVlf3X3/FvWfwfK1ZnZOtrLEvOdxZrbczDYEv/9fiNOm0swaYn6+X812ruB9u/y5hLS+Jsesh9VmttfMvtipTU7Wl5k9YGY7zGxdzLykPocy8rcY7xK9/fUBnA5MJnqP9YqY+WcAa4BBwIlELy9fGKf/t4AFwfMFwDeznPc7wFcTLHsDODaH6+4O4MvdtCkM1t1JwMBgnZ6R5VyXAgOC599M9DPJxfpK5vsnejXqPxC9I+d5wF9y8LMbB5wTPB9G9BYKnXNVAo/l6vcp2Z9LGOsrzs/0HaIn2uV8fQEfAM4B1sXM6/ZzKFN/i9oCieHuG9x9Y5xFc4Eqdz/o7luBzcDMBO0eCp4/BFyRlaBE//MCPgr8IlvvkQUzgc3uvsXdDwFVRNdZ1rj7H929NZh8geidLcOSzPc/F/ipR70AlHbcfTNb3H2bu78cPN8HbAB6y525cr6+OrkIeN3d073CRY+4+zPA7k6zk/kcysjfogpIcsqBt2Oma4n/B1bm7tsg+kcJjMlipvcD2919U4LlDvzRzFaa2bws5og1PxhGeCDBZnOy6zFbPkX0v9V4crG+kvn+Q11HZjYROBv4S5zF7zWzNWb2BzObkqNI3f1cwv6duobE/8SFsb4guc+hjKy3fL6lbVaY2ZPA2DiLvuKJ79duceZl7fjnJDNeS9dbH+e7e52ZjQGeMLNXg/9WspIL+CFwJ9H1cifR4bVPdX6JOH17vB6TWV9m9hWgFfh5gpfJ+PqKFzXOvM7ff05/1454Y7MS4DfAF919b6fFLxMdpmkM9m8tASblIFZ3P5cw19dA4MPAbXEWh7W+kpWR9dbvCoi7X5xGt1rguJjpCUBdnHbbzWycu28LNqN3ZCOjmQ0ArgJmdPEadcHXHWa2mOgma48+EJNdd2b2I+CxOIuSXY8ZzWVm1wMfBC7yYAA4zmtkfH3Fkcz3n5V11B0zKyJaPH7u7o90Xh5bUNx9qZn9wMyOdfesXjgwiZ9LKOsrcBnwsrtv77wgrPUVSOZzKCPrTUNYyXkUuMbMBpnZiUT/k3gxQbvrg+fXA4m2aHrqYuBVd6+Nt9DMhprZsI7nRHckr4vXNlM6jTtfmeD9XgImmdmJwX9v1xBdZ9nMNQe4FfiwuzclaJOr9ZXM9/8o8I/B0UXnAQ0dwxHZEuxP+wmwwd3vSdBmbNAOM5tJ9LNjV5ZzJfNzyfn6ipFwFCCM9RUjmc+hzPwtZvsogd70IPrBVwscBLYDy2KWfYXoUQsbgcti5v+Y4IgtYBTwFLAp+DoySzkfBG7sNG88sDR4fhLRoyrWAOuJDuVke939N1ADrA1+Ecd1zhVMX070KJ/Xc5RrM9Gx3tXB474w11e87x+4sePnSXRo4d5geQ0xRwNmMdPfEB2+WBuzni7vlGt+sG7WED0Y4X05yBX35xL2+gredwjRgjAiZl7O1xfRArYNaAk+u/4p0edQNv4WdSkTERFJi4awREQkLSogIiKSFhUQERFJiwqIiIikRQVERETSogIiEhKLXgV3q5mNDKaPCaZPCDubSDJUQERC4u5vE70EzMJg1kJgkYd0YT6RVOk8EJEQBZcQWQk8AHwaONujV0cVyXv97lpYIvnE3VvM7BbgceBSFQ/pTTSEJRK+y4hejmJq2EFEUqECIhIiMzsLuITo3fRuyvHNkER6RAVEJCTB1Vp/SPT+G28BdwPfDjeVSPJUQETC82ngLXd/Ipj+AXCamV0QYiaRpOkoLBERSYu2QEREJC0qICIikhYVEBERSYsKiIiIpEUFRERE0qICIiIiaVEBERGRtPx/+ccddj2Q3YsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y = (np.exp(2*x) - 1) / (np.exp(2*x) + 1)\n",
    "plot_graph(y, \"TanH(X)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d526d202",
   "metadata": {},
   "source": [
    "### Rectified Linear Units (ReLU)\n",
    "\n",
    "The rectified linear unit (ReLU) [[@teh2001rate]](https://www.cs.toronto.edu/~hinton/absps/nips00-ywt.pdf) was introduced by Teh & Hinton in 2001. The Rectified Linear Unit has become very popular in the last few years. The following equation shows the straightforward ReLU function:\n",
    "\n",
    "\n",
    "$$ f(x) = max(0,x) $$\n",
    "\n",
    "In other words, the activation is simply thresholded at zero\n",
    "\n",
    "$$\n",
    "f(x) =\n",
    "  \\begin{cases}\n",
    "    x       & \\quad \\text{if } x \\geq 0 \\\\\n",
    "    0       & \\quad \\text{if } x < 0\n",
    "  \\end{cases} \n",
    "$$\n",
    "\n",
    "There are several pros and cons to using the ReLUs:\n",
    "\n",
    "- It was found to greatly accelerate (e.g. a factor of 6 [[@krizhevsky2012imagenet]](http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf) ) the convergence of stochastic gradient descent compared to the sigmoid/tanh functions. It is argued that this is due to its linear, non-saturating form.\n",
    "\n",
    "- Compared to tanh/sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.\n",
    "\n",
    "- But the issue is that all the negative values become zero immediately which decreases the ability of the model to fit or train from the data properly. That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turns affects the resulting graph by not mapping the negative values appropriately.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "adf5ad74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAiFElEQVR4nO3de3ycZZ338c8vaZqmSdsU2oY2BQr2ANJiD7GgiKacCjxKoS4IyK6I0kdXbD2ActgVZXXBxVVBFLa6PLgWCLty8sCWc0BRSg8pPQA9AqVp6dGQJk3SHH7PHzPhlSYz7UzmcM/h+3698urMfc+d+zvXJL9eueae6zJ3R0RE8kdB0AFERCS9VPhFRPKMCr+ISJ5R4RcRyTMq/CIieWZA0AFiMWLECB83bly/jm1ubqa0tDS5gZJAueKjXPFRrvhkai5ILNvy5ct3u/vIPjvcPeO/ZsyY4f31/PPP9/vYVFKu+ChXfJQrPpmayz2xbMAyj1BTNdQjIpJnVPhFRPKMCr+ISJ5R4RcRyTMq/CIieSZlhd/M7jWznWa2pse2I8zsaTPbEP53eKrOLyKSzR6rq+e0257jysXNnHbbczxWV5+0753KHv99wLm9tl0PPOvuE4Bnw/dFRKSHx+rqueGR1dQ3tABQ39DCDY+sTlrxT1nhd/cXgb29Ns8Bfh2+/WvgwlSdX0QkW93+5Dpa2jsP2tbS3sntT65Lyvc3T+F8/GY2DviDu08O329w9/Ie+//m7hGHe8xsHjAPoKKiYkZNTU2/MjQ1NVFWVtavY1NJueKjXPFRrvhkWq4rFzdH3XffubF/infWrFnL3b2q9/aMnbLB3RcCCwGqqqq8urq6X9+ntraW/h6bSsoVH+WKj3LFJ9Nylb/wFA0t7X22V5aXJCVnuq/q2WFmowHC/+5M8/lFRDLamvr32NfaToEdvL2kqJDrZk9KyjnSXfh/B3wufPtzwONpPr+ISEYrLDA+fNwR3DLnJCrLS4BQT//WuVO4cFplUs6RsqEeM3sQqAZGmNlW4GbgNuC/zewLwBbg4lSdX0Qkm7g7ZsaJo4dSM+8jAFxx6riUDEOlrPC7+2VRdp2ZqnOKiGSre17YzJa9+7llzkkUFaZ2MEaf3BURCdhfN+3h9iffoLG1nQG9B/dTQIVfRCRAOxtb+eqDdYwbUcoPP30yZqkv/Bl7OaeISK5r7+zimgfqaG7r4IGrT6GsOD0lWT1+EZGArN+xj7Xb3uPWuVOYWDEkbedVj19EJCAnjRnGC9+axYiy4rSeVz1+EZE0e3N3M4tefht3T3vRB/X4RUTSquVAJ19etJx3G1s5b/JRHKnCLyKSu9ydf3psDet27OP/XfnhQIo+aKhHRCRtapa+w8MrtjL/jAlUTxoVWA4VfhGRNNi1r43v/X4tp08YwfwzJwSaRUM9IiJpMHJIMfdcMYOTx5ZTmIZP5x6KCr+ISAo8VlfP7U+uY1tDCyOHFHPj+ScmbXbNRKnwi4gkWfeaud3LJ+7c18a3H14FkBHFX2P8IiJJFmnN3LaOrqStmZsoFX4RkSTb1tAS1/Z0U+EXEUmy0eWDIm4fE15RK2gq/CIiSXbd2ZP6zKufzDVzE6U3d0VEkqi9s4uLZozFCuz9q3rGlJdw3exJGfHGLqjwi4gkzeZdTfzDva/w40umcuG0yowp9L1pqEdEJAn2H+jgy4tW0NzWQeXwzBjLj0Y9fhGRBLk7//ToGtbv3MevPz+Tygx5Ezca9fhFRBL0wCtbeKSungVnTuDjE0cGHeewVPhFRBJUt6WBT0wcyfwzgp18LVYa6hERSdDtf3cybR1dFAQ8+Vqs1OMXEemHri7nlt+/xuZdTZgZg4oKg44UMxV+EZF++PnzG7n3pTf5y6Y9QUeJmwq/iEic/rxhNz9+Zj1zpo7hs6ccE3ScuKnwi4jEYft7LcyvqWP8yDJunTsFs+wY1+9JhV9EJA53PruBtvZO7r5iBoMHZuf1MdmZWkQkIDd/6iQuqTqa8aPKgo7Sb+rxi4jEYOlbe2lsbWdQUSHTjhkedJyEBNLjN7OvA18EHFgNfN7dW4PIIiISTc91cwGmH1POw/94WsCpEpf2Hr+ZVQLzgSp3nwwUApemO4eIyKF0r5tb39CCE+qlrt3WyGN19UFHS1hQQz0DgBIzGwAMBrYFlENEJKJI6+a2ZtC6uYkwd0//Sc0WAD8AWoCn3P2zER4zD5gHUFFRMaOmpqZf52pqaqKsLPPehFGu+ChXfJQrPpFyXbm4Oerj7zu3NNWR3pdIm82aNWu5u1f13p72MX4zGw7MAY4DGoD/MbMr3H1Rz8e5+0JgIUBVVZVXV1f363y1tbX099hUUq74KFd8lCs+kXJVvvwc9REWR68sL0nrc0hFmwUx1HMW8Ka773L3duAR4KMB5BARiai5rYNrz55ISa/5dzJp3dxEBFH4twCnmtlgC33k7Uzg9QByiIj00dXlfOWBFTyx9l3+9aLJVJaXYIR6+rfOnZKxyynGI+1DPe6+xMx+C6wAOoA6wkM6IiJBu+v5jdSu28X3L5zMRdPHctH0sUFHSrpAruN395uBm4M4t4hINC+u38VPnlnPRdMqs3LytVjpk7siIsC2hhYW1NQxYVQZP7hoclZOvhYrFX4REWBv8wGOLCvO6snXYpXbz05EJEaTK4fx1Nc+njXLJyZCPX4RyWu/f3UbNW+00d6ZPWvmJko9fhHJWxt37uPbD6+icnD6ZzAIkgq/iOSl5rYOvrRoBYMHFvKPUwspKsyfAZD8eaYiImHuzg2PrGbzribuvHQawwflVynMr2crIgJs2tXE4rXv8s1zJvHR8SOCjpN2GuoRkbwzftQQFi84nXFHpm+WzUyiHr+I5I29zQf4w6rQ8h/HjyzLm6t4elOPX0TyQmeXs6CmjiVv7mXaMcOpLC8JOlJgVPhFJKd1r5vbPbf+xVVj87rog4Z6RCSH9Vw3t9sfXt2WE+vmJkKFX0RyVqR1c1vac2Pd3ESo8ItIztoWYenEQ23PFyr8IpKzRg0tjrh9jMb4RURyz+Mr69m9r43iAQeXuVxZNzcRKvwiknM27NjH9Q+vZvqxw3N23dxE6HJOEckpTW0dfGnRckqLC7nr8ulUDB3Ep2ccHXSsjKLCLyI5w925/uFVvLm7mUVfPIWKoYOCjpSRNNQjIjnDHY49cjDXzp7ERz+Qf5OvxUo9fhHJCe5OQYFx3ewTgo6S8dTjF5Gst6epjQt/8ReWv7036ChZQYVfRLJaaPK1lby+vZHiAYVBx8kKKvwiktXueGY9f964m1suOInJlcOCjpMVVPhFJGs9v24ndz63kb+bMZbPfFiXbMZKhV9EstbvV27jhKOG8C9zJmOWn4uq9Ieu6hGRrPWjiz9EQ0s7JQM1th8P9fhFJOv86k+bqW9ooaDAOKJ0YNBxso4Kv4hklcfq6vn+H1/noVe2BB0lawUy1GNm5cCvgMmAA1e5+1+DyCIima97+cTuefSPH1HK/DMnBJwqewXV478DWOzuJwAfAl4PKIeIZLieyyc6oZ7itoYW/rBqe9DRslbaC7+ZDQU+DvwngLsfcPeGdOcQkewQafnE1g4tn5gIc/f0ntBsKrAQeI1Qb385sMDdm3s9bh4wD6CiomJGTU1Nv87X1NREWVlZIpFTQrnio1zxyaVcVy5ujrrvvnNLE40EZG57QWLZZs2atdzdq3pvD6LwVwEvA6e5+xIzuwNodPd/jnZMVVWVL1u2rF/nq62tpbq6ul/HppJyxUe54pNLuWb+4Bl27mvrs72yvISXrj8jsFzpkkg2M4tY+IMY498KbHX3JeH7vwWmB5BDRDLc7qY2Wts7Kej12Swtn5iYtBd+d38XeMfMul+1MwkN+4iIvC80+VodbR1dfOOciVo+MYliupzTzAYBnwROB8YALcAa4I/uvrYf5/0qcL+ZDQQ2A5/vx/cQkRz2k6fX89LGPfzbp0/mkg8fzTWzdPlmshy28JvZd4FPAbXAEmAnMAiYCNwW/k/hm+6+KtaTuvtKoM+4k4gIwHNv7OCu5zdySdVYLtHka0kXS49/qbt/N8q+H5vZKOCY5EUSkXx3/IgyLppWyS1zJgcdJSfFUvj/N9oOMyt3952E/goQEUlIe2cXAwqMcSNK+clnpgYdJ2fF8ubuMjM7pfdGM/sisCL5kUQkX33n8bVc80AdXV3pvcw838RS+OcDC83sl2Z2hJlNM7O/ArMJfQJXRCRhDy/fyoOvbOHoIwZT0Pv6TUmqww71uPufzWw68D1gE9AEfMHdn0p1OBHJD2+828hNj63m1OOP4NpzJgYdJ+fFeh3/xcBlwN3AduAzZnZEylKJSN7Y19rOlxetYOigIu68bBoDCjVbfKodtoXN7Bngs8BZ7n4jcAqwElgank9HRKTf3t6zn6a2Du66fDqjhgwKOk5eiOW/1p+7+6fc/U0AD/kZcBrwiZSmE5GcN7lyGH/61ixmHqdBhHQ5bOF390ejbH/X3T+b/Egikg+WvrWXnz27ga4uZ1CR1sxNp1iGen5vZp8ys6II+443s1vM7KrUxBORXLRrXxtfuX8FD6/Yyv5ec+1L6sXyAa6rgW8APzWzvcAuQlM2jCN0lc9d7v54yhKKSE7p6Oxi/oN1NLa28+urZlJWHMgKsHktlss53wW+BXzLzMYBowlN0rbe3fenNp6I5ILuNXPrG1ooe/5pmto6+NHFH+LE0UODjpaX4vqv1t3fAt5KSRIRyUnda+Z2L5/Y1NZBYYExQB/SCkwsY/z7zKyxx9d7ZrbJzH5lZkemI6SIZK9Ia+Z2drnWzA1QLFf1DHH3oT2+hhGaUnktcE/KE4pIVtvW0BLXdkm9fn1Ezt3/5u4/AT6Q5DwikmPGlJfEtV1Sr9+fjQ5f3qm340XkkD4+cUSfbVozN1ixrMA1N8Lm4cBnCC2ULiIS0WvbGnlkRT3jR5ayv72TbQ2tVJaXcN3sSVozN0Cx9Ng/1eu+A3uAO9z9j8mPJCK5oOVAJ/94/3KGlRTx4LyPMHJIMbW1tVRXVwcdLe/Fch1/1IXQzWyLu2vZRRHpY1BRAf/3Ex9g/KgyRg4pDjqO9JDoGL0uxBWRPhpb2xk6qIjLZqpfmIkSnfha66OJyEGWbN7Dabc9x5LNe4KOIlHE8ubuN6LtAsqSG0dEstnOfa1c82AdI8uK+eAYTceQqWIZ6hlyiH13JCuIiGS3js4uvvpAHfta2/nNF2YyZFCfCX0lQ8Ty5u730hFERLLbj55az5I39/LjSz7ECUept5/JYh7jN7OJZvasma0J3z/ZzP4pddFEJFu4Oy0HOrj8lGOYO31s0HHkMOK5queXwHXAfwC4+yozewD4fiqCiUj2MDO+N2cyXV263iMbxHNVz2B3f6XXto5khhGR7NLa3sn//c0yVm99D4ACTbWcFeIp/LvN7AOEL+E0s78DtqcklYhkhZsfX8uTa3ewq6k16CgSh3iGer4CLAROMLN64E1Ai62L5Kn/XvYODy17h2tmjeeMEyqCjiNxiLnwu/tm4CwzKyX0l0ILoYna3k5RNhHJUGu3vcc/P7aG08YfydfPnhh0HIlTLB/gGkqot18JPA48E75/LfAqcH9/TmxmhcAyoN7dP9mf7yEi6dO9bu62hhZKBhZSPKCAOy6dRqHG9bNOLGP8vwEmAauBq4GngIuBC919TgLnXgC8nsDxIpIm3evm1je04MD+A520dXTx5w27g44m/RBL4T/e3a909/8ALiO07OIn3X1lf09qZmOB/wP8qr/fQ0TSJ9K6uW0dXVo3N0uZ+6GvuzWzFe4+Pdr9fp3U7LfArYSmg7g20lCPmc0D5gFUVFTMqKmp6de5mpqaKCvLvCmFlCs+yhWfZOe6cnFz1H33nVsa8/fJl/ZKpkSyzZo1a7m7V/XeHsubux8ys8bwbQNKwvcNcHeP67PZZvZJYKe7Lzez6miPc/eFhK4ioqqqyvu7eEOmLvygXPFRrvgkO9dRf3mWdxv7XrJZWV4S13nypb2SKRXZYpmrpzCpZ4TTgAvM7HxgEDDUzBa5+xVJPo+IJEFHZxeDB/YtA1o3N3slOh9/3Nz9Bncf6+7jgEuB51T0RTLXPS9sYvPuZq449Rgqy0swQj39W+dO0bq5WSrRFbhEJMd99pRjGV46kM+ecmzQUSRJ0t7j78nda3UNv0hm2tnYyoGOLhX9HBRo4ReRzNRyoJN/uPcVvrRoedBRJAVU+EWkj+88voZ1O/bx9x9RTz8XqfCLyEEeWrqF/1m+la/OGs+sSaOCjiMpoMIvIu9bU/8e//z4Wk6fMIIFZ2nytVylwi8i7xs4oIAPjxvOTz8zVZOv5TBdzikiuDtmxsSKIdz/xVODjiMpph6/iHDPC5v51m9fpb2zK+gokgYq/CJ57i+bdnP7k2/QfKCTARreyQsq/CJ5bEdjK/MfrOO4EaX88NMnY6bCnw80xi+Sp9o7u7jmgRXsP9DJg1efSlmxykG+UI9fJE9t2NHEG9v3cevcKUyoGBJ0HEkj/Rcvkkd6rps7pryEb583iTlTNcNmvlHhF8kT3evmdi+hWN/Qwg/++AZlxUWaXjnPaKhHJE9EWje3pb1T6+bmIRV+kTyxraElru2Su1T4RfLEsJKiiNvHlJekOYkETYVfJA/s2tdGU1s7vT+fpXVz85Pe3BXJAyOHFHPf50/hrT3N3F276f2req6bPUlv7OYhFX6RHNbV5by2vZHJlcP42IQRfGzCCK44VYur5DsN9YjksLtf2MQFd/2Z1VvfCzqKZBAVfpEc9dLG3fz7U+v45MljmFw5NOg4kkFU+EVy0N9au5j/YB3Hjyzj1rlTNPmaHERj/CI5pr2zi1+sbKOl3XjoiumUavI16UU/ESI5ZkCBMb1iAPOnn8T4UZp8TfpS4RfJIe2dXRQVFnDecUVUf2hM0HEkQ2mMXyRHbNrVRPXttby8eU/QUSTDqfCL5ID9Bzr48qLltLR3cswRg4OOIxlOhV8ky7k7Nz26hg07m7jj0qmae0cOS4VfJMvdv2QLj9bV87UzJ3L6hJFBx5EsoMIvkuVWb32PT0wcyVfPGB90FMkSab+qx8yOBv4LOAroAha6+x3pziGSrXovn3jtORM5b8poCnpPvSkSRRCXc3YA33T3FWY2BFhuZk+7+2sBZBHJKpGWT7zx0TWYmWbZlJilfajH3be7+4rw7X3A64B+YkVioOUTJRnM3YM7udk44EVgsrs39to3D5gHUFFRMaOmpqZf52hqaqKsrCzBpMmnXPFRrpArFzdH3XffuaXv31Z7xSdTc0Fi2WbNmrXc3at6bw/sk7tmVgY8DHytd9EHcPeFwEKAqqoqr66u7td5amtr6e+xqaRc8VGukKP+8gzvNrb12V5ZXnJQDrVXfDI1F6QmWyBX9ZhZEaGif7+7PxJEBpFsdNzI0j7btHyixCvthd9C88P+J/C6u/843ecXyWb3fX4m3zh7ApXlJRihnv6tc6fojV2JSxBDPacBfw+sNrOV4W03uvsTAWQRyQpLNu/hhNFDGVZSxPwzJzL/zIlBR5IslvbC7+5/BnTBsUiMNu5s4qr7lnLWByu449JpQceRHKBP7opksOa20ORrg4oKueG8E4OOIzlC8/GLZCh358ZHV7NxVxO/ueoUjho2KOhIkiPU4xfJUDVL3+Hxldv4xlkT+diEEUHHkRyiwi+Soc44YRTXzBrPV2Zp8jVJLhV+kQzT1NZBZ5dTMXQQ186epMnXJOlU+EUySFeX85X7V3D1fy0jyOlUJLep8ItkkJ89t5EX1u/izBNHEfqso0jyqfCLZIgX1+/ip8+uZ+60Si6feUzQcSSHqfCLZID6hhYW1NQxcdQQfnDRFPX2JaVU+EUyQMP+A4wcUszdV0ynZGBh0HEkx+kDXCIZ4KQxw1i84OO6gkfSQoVfJADd6+bWN7RQWlzILRecxKdnHB10LMkTKvwiadZ73dzmtk5uemwNhQUFml5Z0kJj/CJpFmnd3Nb2Lq2bK2mjwi+SZtsaWuLaLpJsKvwiaTZySHHE7WPKS9KcRPKVCr9Imt14/okUDzj4V0/r5ko66c1dkTTZ23yAP23Y9f4buLc/uY5tDS2MKS/hutmT9MaupI0Kv0gadHY5C2rqWPLmXmYcO5wLp1Wq0EtgNNQjkgZ3PruBP23YzfcuOImxwwcHHUfynAq/SIrVrtvJnc9t4NPTx3Lph/UhLQmeCr9ICjW2tvP1h1YyqWII379wsiZfk4ygMX6RFBo6qIh/vWgKJ4weqsnXJGOo8IukyNa/7Wfs8MGcN2V00FFEDqKhHpEUeHxlPbN+VMvSt/YGHUWkDxV+kSRbv2Mf1z+8mqlHlzP16PKg44j0ocIvkkRNbR18adFySosHcNfl0ykq1K+YZB79VIokibvz7YdX8dbuZn522TQqhg4KOpJIRCr8IkniDh8YWcZ1s0/gIx84Mug4IlHpqh6RJHB3CgqMb5w9MegoIoelHr9IgnY3tXHBXS/pCh7JGoH0+M3sXOAOoBD4lbvfluxz9FzTtPLl5+Ka/bD72P7OnJjI8UEfq/aKv72Kn3mOjs4uSor0AS3JDmkv/GZWCPwcOBvYCiw1s9+5+2vJOkfvNU3rG1q44ZHVAIf9xU7k2CDPnY3HZmvu3se2dXRRVGhs3NnE5MphhzxWJBOYu6f3hGYfAb7r7rPD928AcPdbox1TVVXly5Yti/kcp932HPURlrEbUGAcN6KUqnHDuXXuyQBc/suX2bWv7f3HvLm7mY6uvm3SfSzA7JOO4trwohnn/vRFOns8PpbjL5t5DMd3vM3UmR/l4nv+GtexANecMZ45UyvZuLOJLy9aHtexADd/6iQ+NmEES9/ay42PrI567JGlAzmidGCf7XdcOo0PjhnK4jXbueaBuojHVpaX8M1zJnJ37aY+++7/4imMGjqI+5e8zc2Pr416/OdPG8dDS98BoHl/M6WDQ8/jiQWnU1RYwJTvPsm+1o6Ix750/Rn8cPEbPPPajoP2DSsp4rdf/mjUn5HiAQWs+/55AHz7t6tYseVvB+0/fmQpa+obIx7bfd5MUFtbS3V1ddAx+lCu+CWSzcyWu3tV7+1BDPVUAu/0uL8VOKX3g8xsHjAPoKKigtra2phPEOmXEqCjyxlmLbT9rY3a2tB4bPGBNoaZH/SYQx0L8N6OLdTWbgdgKK102cGPO9zxO7dsYtTQVv7y0ksMs7a4jgV4e8Pr1DZsYNf+LobZgbiOBVi39lU66gt5u7GTYdYe9dg9zQc4rqyzz/ZVdcvYub6AzXs6ox5b39DClo1vMMz6FuaXX/4rQwca297tOOTxO9/Z/P7xZcVdFIafx4svvEBhgUUs+t3H1tbW0rijnWF2cP6SjlZqa2uj/oy0dXS9/7N2oOEAw6zroP3W3Ep9w6HPmwmampoyJktPyhW/VGQLosd/MTDb3b8Yvv/3wEx3/2q0Y5LV44+lR5bIsfEcH+l/8aByq73Sd2y6ZGoPVrnil4oefxBX9WwFek5KPhbYlswTXDd7Up832mJd0zSRY4M8dzYeG+S5g3zOIkELYqhnKTDBzI4D6oFLgcuTeYKea5rWN7RQGccVG4muh5rI8ZlwrNorte0lkhHcPe1fwPnAemATcNPhHj9jxgzvr+eff77fx6aScsVHueKjXPHJ1FzuiWUDlnmEmhrIdfzu/gTwRBDnFhHJd/rkrohInlHhFxHJMyr8IiJ5RoVfRCTPpP0DXP1hZruAt/t5+AhgdxLjJItyxUe54qNc8cnUXJBYtmPdfWTvjVlR+BNhZss8wifXgqZc8VGu+ChXfDI1F6Qmm4Z6RETyjAq/iEieyYfCvzDoAFEoV3yUKz7KFZ9MzQUpyJbzY/wiInKwfOjxi4hIDyr8IiJ5JicKv5ldbGZrzazLzKp67bvBzDaa2Tozmx3l+CPM7Gkz2xD+d3gKMj5kZivDX2+Z2cooj3vLzFaHHxf76jP9z/VdM6vvke38KI87N9yGG83s+jTkut3M3jCzVWb2qJmVR3lcWtrrcM/fQu4M719lZtNTlaXHOY82s+fN7PXwz/+CCI+pNrP3ery+30l1rvB5D/m6BNRek3q0w0ozazSzr/V6TFray8zuNbOdZramx7aY6lBSfhcjTdmZbV/AicAkoBao6rH9g8CrQDFwHKFpoAsjHP9vwPXh29cDP0xx3n8HvhNl31vAiDS23XeBaw/zmMJw2x0PDAy36QdTnOscYED49g+jvSbpaK9Ynj+hqcb/FzDgVGBJGl670cD08O0hhKY6752rGvhDun6eYn1dgmivCK/pu4Q+4JT29gI+DkwH1vTYdtg6lKzfxZzo8bv76+6+LsKuOUCNu7e5+5vARmBmlMf9Onz718CFKQlKqKcDXAI8mKpzpMBMYKO7b3b3A0ANoTZLGXd/yt27F7d9mdBKbUGJ5fnPAf7LQ14Gys1sdCpDuft2d18Rvr0PeJ3QmtbZIO3t1cuZwCZ37++MAAlx9xeBvb02x1KHkvK7mBOF/xAiLewe6Rejwt23Q+iXCRiVwkynAzvcfUOU/Q48ZWbLwwvOp8M14T+3743y52Ws7ZgqVxHqHUaSjvaK5fkH2kZmNg6YBiyJsPsjZvaqmf2vmZ2UpkiHe12C/pm6lOidryDaC2KrQ0lpt0AWYukPM3sGOCrCrpvc/fFoh0XYlrLrV2PMeBmH7u2f5u7bzGwU8LSZvRHuHaQkF3A38C+E2uVfCA1DXdX7W0Q4NuF2jKW9zOwmoAO4P8q3SXp7RYoaYVvv55/Wn7WDTmxWBjwMfM3dG3vtXkFoOKMp/P7NY8CENMQ63OsSZHsNBC4AboiwO6j2ilVS2i1rCr+7n9WPw2Jd2H2HmY129+3hPzd3piKjmQ0A5gIzDvE9toX/3WlmjxL60y6hQhZr25nZL4E/RNgVazsmNZeZfQ74JHCmhwc4I3yPpLdXBLE8/5S00eGYWRGhon+/uz/Se3/P/wjc/Qkz+4WZjXD3lE5IFsPrEkh7hZ0HrHD3Hb13BNVeYbHUoaS0W64P9fwOuNTMii20uPsE4JUoj/tc+PbngGh/QSTqLOANd98aaaeZlZrZkO7bhN7gXBPpscnSa1z1oijnWwpMMLPjwr2lSwm1WSpznQt8G7jA3fdHeUy62iuW5/874B/CV6ucCrzX/Wd7qoTfL/pP4HV3/3GUxxwVfhxmNpPQ7/yeFOeK5XVJe3v1EPWv7iDaq4dY6lByfhdT/e51Or4IFaytQBuwA3iyx76bCL0Lvg44r8f2XxG+Agg4EngW2BD+94gU5bwP+FKvbWOAJ8K3jyf0Lv2rwFpiWIg+CZl+A6wGVoV/gEb3zhW+fz6hq0Y2pSnXRkJjmSvDX/cE2V6Rnj/wpe7Xk9Cf4D8P719Nj6vLUpjpY4T+zF/Vo53O75XrmnDbvEroTfKPpiFXxNcl6PYKn3cwoUI+rMe2tLcXof94tgPt4dr1hWh1KBW/i5qyQUQkz+T6UI+IiPSiwi8ikmdU+EVE8owKv4hInlHhFxHJMyr8InGy0KyYb5rZEeH7w8P3jw06m0gsVPhF4uTu7xCa6uK28KbbgIUe0IRfIvHSdfwi/RCeKmE5cC9wNTDNQ7MlimS8rJmrRySTuHu7mV0HLAbOUdGXbKKhHpH+O4/Qx+4nBx1EJB4q/CL9YGZTgbMJrR719TQvIiKSEBV+kTiFZ2+8m9D891uA24EfBZtKJHYq/CLxuxrY4u5Ph+//AjjBzD4RYCaRmOmqHhGRPKMev4hInlHhFxHJMyr8IiJ5RoVfRCTPqPCLiOQZFX4RkTyjwi8ikmf+P6ObSuUpI91wAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# def relu(x):\n",
    "#     if x >= 0:\n",
    "#         return x\n",
    "#     else:\n",
    "#         return 0\n",
    "# y = list(map(relu, x))\n",
    "\n",
    "y = list(map(lambda a: a if a>=0 else 0, x))\n",
    "plot_graph(y,\"ReLU(X)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13b028a7",
   "metadata": {},
   "source": [
    "## Varients of ReLU\n",
    "\n",
    "\n",
    "#### Leaky ReLU\n",
    "\n",
    "Leaky ReLUs are one attempt to fix the “dying ReLU” problem. Instead of the function being zero when $x < 0$, a leaky ReLU will instead have a small negative slope (of 0.1, or so). That is, the function computes \n",
    "\n",
    "$$\n",
    "f(x) =\n",
    "  \\begin{cases}\n",
    "    x       & \\quad \\text{if } x \\geq 0 \\\\\n",
    "    \\alpha x       & \\quad \\text{if } x < 0\n",
    "  \\end{cases} \n",
    "$$\n",
    "\n",
    "where α is a small constant. \n",
    "\n",
    "Some people report success with this form of activation function, but the results are not always consistent. \n",
    "\n",
    "#### PReLU\n",
    "\n",
    "The slope in the negative region can also be made into a parameter of each neuron, as seen in PReLU neurons, introduced in Delving Deep into Rectifiers [[@he2015delving]](https://arxiv.org/pdf/1502.01852.pdf) ), by Kaiming He et al., 2015. However, the consistency of the benefit across tasks is presently unclear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "33960a38",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAkZUlEQVR4nO3deXxU9b3/8dcnCYSBAGGRQCKyKIKKCxJXqoW6odXqrVqttcWVa3vbamtr1d5bbf31V3u9tbW1taXqxVbbtApVa624IFprVUAQUaSsCgRBlgAJIevn/jETGsIMzCQzc2Z5Px+PPDJzzsycd84kn3zne875fs3dERGR/FEQdAAREUkvFX4RkTyjwi8ikmdU+EVE8owKv4hInikKOkA8Bg4c6MOHD+/Uc+vq6ujVq1dyAyWBciVGuRKjXInJ1FzQtWzz58/f5O4H7LXC3TP+a/z48d5ZL774Yqefm0rKlRjlSoxyJSZTc7l3LRswz6PUVHX1iIjkGRV+EZE8o8IvIpJnVPhFRPKMCr+ISJ5JWeE3swfNbKOZLW63rL+ZPWdmyyLf+6Vq+yIi2ezxBeuYcOdsrnimjgl3zubxBeuS9tqpbPFPByZ3WHYz8IK7jwJeiNwXEZF2Hl+wjltmvs26mnoA1tXUc8vMt5NW/FNW+N39ZWBLh8XnAw9Fbj8EXJCq7YuIZKu7Zi2lvqllj2X1TS3cNWtpUl7fPIXj8ZvZcOApdx8buV/j7qXt1m9196jdPWY2FZgKUFZWNr6qqqpTGWpraykpKenUc1NJuRKjXIlRrsRkWq4rnqmLuW765Piv4p00adJ8d6/suDxjh2xw92nANIDKykqfOHFip15nzpw5dPa5qaRciVGuxChXYjItV8mLs6htaN5reUVpKCk5031WzwYzGwIQ+b4xzdsXEcloM+avpbahmcIC22N5qFsh3zxrdFK2ke7C/yQwJXJ7CvBEmrcvIpKx3li1hZtnLuLkgwfw3xceSUVpCAi39H/w6SO5YFxFUraTsq4eM/s9MBEYaGZrgduAO4E/mtnVwAfAxanavohINlm9qY5//+08hvbvyX2fG0/fnt24cPzQlHRDpazwu/tnY6w6LVXbFBHJVqs21xHqVsiDU46jb89uKd1Wxh7cFRHJJ5NGD+LFb06kuKgw5dvSkA0iIgFxd77zxGL+MPcDgLQUfVDhFxEJzAOvrOI3/3if9zfvTOt2VfhFRALw3Lsb+P7TSzjnyMF848zknKYZLxV+EZE0e6d6G9dXLeCoir786OJjKOhwzn6qqfCLiKTZayu3UBrqxq+/UEmoe3r69dvTWT0iIml29cdG8JnKA+ndI7WnbcaiFr+ISBq0tjq3zFzE3NXhQYuDKvqgwi8ikhZ3PbuU37+xhkVrtwUdRV09IiKp8PiCddw1aynVNfX07dmNmp1NXHbCQVw1YXjQ0VT4RUSSrW0GrbbJVGp2NlFgMP6gUszSewZPNOrqERFJsmgzaLU63P3csoAS7UmFX0Qkyaojc+XGuzzdVPhFRJJsSN8eUZeXR8bXD5oKv4hIErk7B/XvudfyZM6g1VUq/CIiSfTrv63ktVVbOOOwQVSUhjCSP4NWV+msHhGRJJn1zof84K/v8ckjh/Czz45L+xg88VKLX0QkCdydaS+v5KgDS/nRZ47O2KIPavGLiCSFmfHQVcezq6mFHt3SP/BaItTiFxHpgp2NzfzwmffY2dhMSXERA0uKg460Xyr8IiKd1Nrq3FC1kF+9tIKFa2qCjhM3FX4RkU764TPv8ey7G/jPTx7OyQcPDDpO3FT4RUQ6oeqND/jVyyu5/MSDuDIDBl5LhAq/iEiC6htbuPu5f3LKqIHcft4RGTHwWiJ0Vo+ISIJC3Qt57LqTKe3VjaLC7Gs/Z19iEZGAbK1r5MFXVoWHZRjQkz4BzqLVFWrxi4jEobG5lesens+CNTWceugBHDKoJOhInRZI4TezrwHXAA68DVzp7ruCyCIiEkv7WbRC3QvZ2djCPZcek9VFHwLo6jGzCuCrQKW7jwUKgUvTnUNEZF/aZtFaV1OPAzsbWygqMNyDTtZ1QfXxFwEhMysCegLVAeUQEYkq2ixaza3OXbOWBpQoecwD+PdlZtcD3wfqgWfd/XNRHjMVmApQVlY2vqqqqlPbqq2tpaQk8z6WKVdilCsxypWYaLmueKYu5uOnT+6V6ki7dWWfTZo0ab67V3ZcnvY+fjPrB5wPjABqgEfN7HJ3f7j949x9GjANoLKy0idOnNip7c2ZM4fOPjeVlCsxypUY5UpMtFyDX32eD7c37PXYitJQWn+GVOyzILp6TgdWuftH7t4EzARODiCHiEhUdQ3NFES5KCuTZtHqiiAK/wfAiWbW08KXu50GLAkgh4jIXlpaneurFvLh9l1MPXVExs6i1RVp7+px99fN7DHgTaAZWECkS0dEJGh3/nUJzy/ZwHc/dQRTTh7OreccHnSkpAvkPH53vw24LYhti4jEMv/9Lfz6b6uYctIwppw8POg4KaMrd0VEIsYP68+0z4/nE2MGBR0lpTRWj4jkvfW1rbxbvR2AM48YnJUDryUit386EZH92FLXyI/f3MV1D8+nuaU16Dhpoa4eEclbDc0tXPfb+WzZ5fxhyjE539Jvkx8/pYhIB+7OrTMX88bqLVwztpjxw/oFHSlt1OIXkbz0xMJqZry5lhtOH8UxRfk1XJgKv4jkpU8eNYTG5lYurjyQl17Kr8Kvrh4RySvvVm9nU20D3QoL+MxxQ7NuvtxkUOEXkbyxrqaeLzz4Bl/53YKgowRKXT0iktPaz6JVWGAUGnz3/COCjhUotfhFJGd1nEWrudVxbPfFWvlKhV9Ecla0WbQaW1pzYhatrlDhF5GcVV1Tn9DyfKHCLyI564DexVGXl5eG0pwks6jwi0hOWr5xB9vqGyks2PN0zVyZRasrVPhFJOdsrm3gyulz6d2jO98+Z0xOzqLVFTqdU0RySkNzC9c9PJ+N2xuomnoi4w7qx1UfGxl0rIyiwi8iOeW7f36Xuau3cu9l4xh3UP4MvJYIFX4RySlfOGkYo8t6c+5R5UFHyVgq/CKSE1ZvqmPYgJ6MGdyHMYP7BB0no+ngrohkvQUfbOWsn7zM//59ddBRsoIKv4hktbVbd3Ltb+ZT1qcH5x+j7p14qKtHRLJWbUMz1zw0j4bmFqqmnsCAkugXbMmeVPhFJCu5OzdULWDZxlqmX3kchwzqHXSkrKHCLyJZycy4uHIopx9WximjDgg6TlZR4ReRrLNx+y4G9enBWUcMDjpKVtLBXRHJKi//8yM+9t8v8uJ7G4OOkrUCafGbWSlwPzAWcOAqd/9HEFlEJPO1n0ULYHCfYo4b0T/gVNkrqBb/PcAz7j4GOBpYElAOEclwHWfRcmDLziaef3dD0NGyVtoLv5n1AU4FHgBw90Z3r0l3DhHJDtFm0Wpo1ixaXWHunt4Nmh0DTAPeJdzanw9c7+51HR43FZgKUFZWNr6qqqpT26utraWkpKQrkVNCuRKjXInJpVxXPFMXc930yb26GgnI3P0FXcs2adKk+e5e2XF5EIW/EngNmODur5vZPcB2d/+vWM+prKz0efPmdWp7c+bMYeLEiZ16biopV2KUKzG5lOukH7zA+m279lpeURri7zd/IrBc6dKVbGYWtfAH0ce/Fljr7q9H7j8GHBtADhHJcE++Vc32+iaKi/YsVZpFq2vSXvjd/UNgjZm1vWunEe72ERHZbf77W/nGo29xRHlfvn/BWM2ilURBXcD1FeARM+sOrASuDCiHiGSgNVt2MvU38xjStwe//Px4+vfqzkWVQ4OOlTMCKfzuvhDYq99JRGTHriaufmguTS2tPDDlOPr36h50pJyjIRtEJKMUFRQwZnAfbjvvCA4ZlJln2mQ7FX4RyRi7mloIdS/kp58dF3SUnKaxekQkIzz06mrOv/fvbK5tCDpKzlPhF5HAzVm6ke/++R2G9u9JaU/16aeaCr+IBGrphzv48u8WMGZwH+659BgKCyzoSDkvrj5+M+sBnAucApQD9cBi4C/u/k7q4olILvtoRwNXTZ9Lz+6FPHBFJb2KddgxHfa7l83sduA8YA7wOrAR6AEcCtwZ+adwo7svSl1MEclFLa3O4L49uP28IxjSNxR0nLwRz7/Xue5+e4x1d5vZIOCg5EUSkVzn7rQ6DO7bg8euOwkzde+kUzx9/H+NtcLMSt19o7t3bgQ1EclLP3l+Gdc9PJ+G5hYV/QDE0+KfZ2ZfbDeoGgBmdg1wKzAyJclEJGe0zaC1rqaefi8/y9adTVw8/kC6F+r8kiDEU/i/CkwzszeAbwHDgF8QHmXz1BRmE5Ec0DaDVttkKlt3NlFgcMKI/mrtB2S//27d/RXCwyZvAFYATwK3ufvF7r42xflEJMtFm0Gr1eHHzy8LKJHE+znrYuCzwH3AeuASM9NMxyKyX20TpMe7XFJvv4XfzJ4HPgec7u63AicAC4G5kekRRURiKi/tEWO5Tt8MSjwt/p+7+3nuvgrAw34GTAA+ntJ0IpLV3J3hA3pR1OFqXM2gFax4+vj/FGP5h+7+ueRHEpFcMf3V1fx9xWZOHTWQikgLXzNoBS+eK3f/DEwDnnH3pg7rRgJXAKvd/cGUJBSRrDT7vQ3c8dS7nHl4Gb+8fDwFBZbRk5rnk3hO57wW+DrwEzPbAnxEeMiG4YTP8rnX3Z9IWUIRyTpL1m/nK79bwGFD+vCTS4+hQAOvZZT9Fv7I5Og3ATeZ2XBgCOFB2v7p7jtTG09EstGKj2rp16s7D0w5jp7dNfBapknoHXH31cDqlCQRkZxx7lHlnHF4GcVFhUFHkSjiOZ1zh5ltb/e1zcxWmNn9ZjYgHSFFJPO1tjo3/vEtnlpUDaCin8HiOaunt7v3affVF6gE3gF+mfKEIpIVfvz8P5nx5lrWbdWFWZmuUyMkuftWd/8xcHCS84hIFpr55lp+Nns5l1QOZeqpGrcx03V6aDwz60aCxwhEJPfMXb2Fm2e8zUkjB3DHBWM18FoWiOc8/k9HWdwPuAR4LOmJRCSrvLJsEwf2C3Hf5cfSvUjDLGeDeFrs53W478Bm4B53/0vyI4lINvnaGYdy9Skj6NOjW9BRJE7xnMd/Zax1ZvaBu2vaRZE809TSyrdmLOKqCSMYW9FXRT/LdLWPvtOdeWZWCMwD1rn7uV3MISIp1jaLVnVNPaHuhexsbOHkgwcytqJv0NEkQV0t/N6F514PLAH6dDGDiKRYx1m0dja2UFRge426KdkhnoO7X4+1CijpzEbN7EDgk8D3CY8DJCIZLNosWs2tzl2zlmqUzSxk7vtutJvZbfta7+7fTXijZo8BPwB6A9+I1tUTmeRlKkBZWdn4qqqqRDcDQG1tLSUlnfr/lFLKlRjlSkyyc13xTF3MddMn94r7dfJlfyVTV7JNmjRpvrtXdlwez8HdhAv7vpjZucBGd59vZhP3sd1phIeDprKy0js7lGumDgOrXIlRrsQkO1fFa7NZF2WqxIrSUELbyZf9lUypyBb3SbdmdqiZvWBmiyP3jzKz/+zENicAnzKz1UAV8Akze7gTryMiaVDf2MJBA0L06HCOvmbRyl6JXG3xa+AWoAnA3RcBlya6QXe/xd0PdPfhkefPdvfLE30dEUm91lbnxkcX8trKLXzhpGFUlIYwNItWtkvkrJ6e7v5Gh8uxm5OcR0QyyI+eW8rTb3/It885jGtPHcmtnzw86EiSBIkU/k1mdjCRUzjN7CJgfVc27u5zgDldeQ0RSY0Z89fy8xdXcOlxQ7nmlBFBx5EkSqTw/wfhg61jzGwdsArQZOsiOai5pZVfvbyCkw/WwGu5KO7C7+4rgdPNrBfhYwP1hAdqez9F2UQkIEWFBfxh6kkUmNGtUAOv5Zp4ZuDqY2a3mNm9ZnYGsBOYAiwHPpPqgCKSPtt2NnHXrPdoaG6hX6/u9O2pMXhyUTwt/t8CW4F/ANcSnni9O3CBuy9MXTQRSaemlla++Mh85q7ewllHDOaoA0uDjiQpEk/hH+nuRwKY2f3AJuAgd9+R0mQikjbuzneeWMyrKzbzPxcfraKf4+LpvGtqu+HuLcAqFX2R3HL/31bx+zfW8KWJB3PR+AODjiMpFk+L/2gz2x65bUAoct8Ad3eNrimSxWp2NvKz2cs4e+xgvnGmrsTNB/GM1VOYjiAiEozSnt2Z+aUJVJSGKNAwy3lB52mJ5KkN23fx8Gvv4+4cMqiEUHe18fJFVydiEZEs0n4WraJCw4BPjBlEeWko6GiSRmrxi+SJtlm01tXU40BTiwPGG6u2BB1N0kyFXyRPRJtFq7GllbtmLQ0okQRFhV8kT1RHmUhlX8sld6nwi+SJWP346t/PPyr8Inlg1aY6zjt6CKFue565o1m08pMKv0iOq9nZyNXT5zLjzXV891NHaBYt0emcIrmssbmVLz78Jmu31vPItSdw3PD+fOa4oUHHkoCp8IvkKHfnvx5fzD9WbubHlxzNccP7Bx1JMoS6ekRy1ItLN/KHeWv4yicO4d/GaeA1+Re1+EVy1KTRg/jl5cdy5uGDg44iGUaFXyTHvFO9jQ/rWjEzJo8dEnQcyUDq6hHJIR9u28VV0+fyi4UNuHvQcSRDqfCL5Iidjc1c/dBcanc1c+1RxZhpiGWJToVfJAe0tjo3VC1kyfrt3HvZsQztrT9tiU2/HSI54Levvc+z727gv849nEljBgUdRzKcDu6K5IBLjhtKz+6Fmi9X4qIWv0gWW7S2hm31TfToVsjFlUPVry9xSXuL38yGAr8BBgOtwDR3vyfdOUSyVftZtDA4srwPT37llKBjSRYJosXfDNzo7ocBJwL/YWaHB5BDJOt0nEXLHZZuqOXxBeuCjiZZJO2F393Xu/ubkds7gCWAhgcUiUO0WbQamjWLliTGgrzIw8yGAy8DY919e4d1U4GpAGVlZeOrqqo6tY3a2lpKSkq6mDT5lCsxyhV2xTN1MddNn9xr923tr8Rkai7oWrZJkybNd/fKjssDO6vHzEqAGcANHYs+gLtPA6YBVFZW+sSJEzu1nTlz5tDZ56aSciVGucIqXpvNuihTJVaUhvbIof2VmEzNBanJFshZPWbWjXDRf8TdZwaRQSTbvFO9jRvPOFSzaEmXpb3wW/h8sweAJe5+d7q3L5KNFq2t4cL7XmX5R7X84NNHahYt6ZIgunomAJ8H3jazhZFlt7r70wFkEcl467fVc81D8xjQq5grJ4zggN7FKvTSJWkv/O7+CqCrTETiUNfQzNXT57GzsYUZXzyBA3oXBx1JcoCu3BXJYDfNWMR7H27n3svGMXpw76DjSI7QWD0iGeyaj43g46MOYOJoDbwmyaPCL5KBVm+qY/jAXow7qB/jDuoXdBzJMerqEckwf1++idPvfokZ89cGHUVylAq/SAZZvrGW6x6ez8gDenHmEWVBx5EcpcIvkiG21DVy9UNzKS4q4IEpx9G7R7egI0mOUh+/SAZobXWue3g+67ft4vfXnsjQ/j2DjiQ5TIVfJAMUFBiXVA7l8hOHMX6YDuZKaqnwiwRs4/ZdDOrTgws1baKkiQq/SADaZtFqG2nzq6cdwtfP0EBrkh46uCuSZu1n0Woz7eWVmkVL0kaFXyTNos2itatJs2hJ+qjwi6RZdZSJVPa1XCTZVPhF0mxIaY+oy8tLQ2lOIvlKhV8kjbbvauKms8ZoFi0JlAq/SJo88vr7nP6jlxg/rJ9m0ZJA6XROkTT427KP+M4T73DqqIGUl4YY2r+nCr0ERi1+kRRbvnEHX3rkTUYNKuFnlx1LYYEmoJNgqfCLpNDm2gaunD6X4qJC7p9SSUmxPmRL8PRbKJJC3YoKOGxwH7448WAO7KeB1yQzqPCLpIC709DcSp8e3Zj2hcqg44jsQV09Iilw7+zlXDLtNXbsago6isheVPhFkuypRdX86Ll/MnJgL/XpS0ZS4RdJogUfbOXGP75F5bB+3HnhkZjpDB7JPCr8IkmydutOrv3NPMr69OBXnx9PcVHh/p8kEgAVfpEkaW5xKvr15MErKhlQUhx0HJGY1AEp0kUtrU6BwfCBvXj8Syere0cyXiCF38wmA/cAhcD97n5nEDlEOqv9DFq9XpzFYYN784d/P1lX5UpWSHtXj5kVAj8HzgYOBz5rZoenO4dIZ3WcQauuoYUFa7bx57eqA04mEp8g+viPB5a7+0p3bwSqgPMDyCF57vEF65hw52xG3PwXJtw5O+bUh00trazdupPlG3cA0WfQaml1zaAlWcPcPb0bNLsImOzu10Tufx44wd2/3OFxU4GpAGVlZeOrqqo6tb3a2lpKSkq6FjoFlCsxyc71anUT0xc30tj6r2VFBXDV2O6cXN6NGcsaeW9zC5t3OVt3OQ6M6FPAbSeHuOKZupivO31yr6Rl7Ip8eR+TJVNzQdeyTZo0ab6773XpeBB9/NE6Qff67+Pu04BpAJWVlT5x4sRObWzOnDl09rmppFyJiZWrra+9uqae8tIQ3zxr9B7DHTe3tLJxRwPVNfV8tKOBs48cAsBXb5+1R9EHaG6Fv3xQyK2XTWT2tsUM9FqOKg1RUdqD8tIQwwf24sSRA6h4bfYeE6W3qSgNZcy+y7b3MWiZmgtSky2Iwr8WGNru/oGAOkfz2P6K976ed/PMRexqClfwdTX13DxzEQCb6xp58JVVfLh9Fy2t/2pXLP1/kykuKmT7ruaor9k27+33zh8bc7vfPGs0t8x8e4/uHs2gJdkkiMI/FxhlZiOAdcClwGUB5JAM0HagtK2Irqup55aZbwNw3tHlfLSjgXU19by+vpn3XlrBReMPZGBJMX+ct4ZvPbZor4+Ku5pauWvWUm4+ewzHj+hPRWmI8tIQ5aU9qCgNUVQQPqxVURqK2mqPZ97btn9KbWf1VCTwz0okE6S98Lt7s5l9GZhF+HTOB939nXTnkOTpbIu9rqGZ///0kr0OlNY3tfC9p97lm4+9RVNLu9L+1nuMG1rKwJJihg/otXf/YER1TT3nHV3OeUeXx9x2V1vtF4yr4IJxFRndRSASSyDn8bv708DTQWxbkitWi91bnQmjBrK2pp7q3V+7OHvsYE4YOYD572/lwvtejfm6W+sa+eLEgykvDVFRGqJ62WI+dcYp9O7RDWB3az4ZrfZE/2GJZDtduSvAnhckVbw2e79FsLXVWbW5jjueejdqi/3OWe+x4dGGPZaXFBcxenBvThg5gBEDe3HT5NH8+uWVbN2599DF5aUhbpo8Zvf9OR8W7C76bZLVahfJNyr8ErPV/sGWOg4Z1JvqmnrWRVrtxx7Uj3//+MG0uHP63S8R62zgjdsbuOOCsbvPiCkvDdGnXeHu36s7X5p4COV9Q50u3mq1i3SOCn8OSbSvfe7qLaz6qI47/hK91f6T55fRdkJMz+6FVJSGOGxIHwC6FRbw88uO5TtPLGZTbeNer11eGuLzJw7bb+auFm+12kUSp8KfI6K12m+asYg5SzcyoKR4d6u9b6gbv736BAC+/5clLFxTE/M13eHpr55CRWmIPqGivQYfO+fIITQ2t3b51EYVb5H0UuHPMPtrtTc0t+we5/3VFZuYu2or1TX1PL5wHQ3Ne16R1NjcyuMLqwl1K6SiX7i7ZXTZv64AvOuio+jRrZBLfvUPqrft2itLeWmIw8v77DOvultEso8Kfwp09vTGP725llv+9PYeFyTd+Ohb/O/fV4IVUF1Tz6baBpZ8bzI9uhXywpKNPPDKKg7oXbxX0W9jwLvfOyvqUMGjynoDcNPkMTpIKpJHVPiTbF8XJJ1z5BA2bN/FwMgkHW9+sJVH561h7dbwgdOVH9XtdW56S6vzTvUOTjp4AGNGD6K8NERzpOP9htNHcdPk0RQXFTLhzujDCJSXhvY7PrwuSBLJLyr8UXSmxe7ubKtv4vsxLkj6+h8X8rU/LsQdfndtuI/9w227eO7djVSU9uDQst6s+Cj64F8trb67X7699qc36oIkEYmXCn8HsVrsLa2tHD9iANU19QzpG+KgAT1Zs2Un//n44t0XKNU1tsR83VaH608bRUW/EAcfUMKSNeFPAOdEBg0D9tlq3x/1tYtIvHK28Cd6QdK2+iaqa+pjXpB046OLdt//xpmH8uVPjKK4qIAtdY2MPKAXHxs1kIrSEL94cTlbolyQVFEa4mtnHLr7/pIoGXRBkoikQ04W/mit9m/NWMTKTbWMHFhCac9uTBw9CHfnkz99hTVbdrKjIfpoje398MIjI2fGhA+KDurTgz9/5WN7PGZgSbEuSBKRjJaThT/aDEkNza389IXlAEwafQATRw/CzDiyoi/Hj+gfGb2xJ7c/+Q4f1Tbs9ZoVpSEuOe6g/W5bFySJSKbLycJfHaWfvM3zXz+VIX3/1Wf+w4uO2mN9U4suSBKR3JaThb88xqiNFaUhDhnUe5/PVXeLiOS6nCz8OkgqIhJbThZ+XZAkIhJbThZ+0AVJIiKxFAQdQERE0kuFX0Qkz6jwi4jkGRV+EZE8o8IvIpJnzGPNlp1BzOwj4P1OPn0gsCmJcZJFuRKjXIlRrsRkai7oWrZh7n5Ax4VZUfi7wszmuXtl0Dk6Uq7EKFdilCsxmZoLUpNNXT0iInlGhV9EJM/kQ+GfFnSAGJQrMcqVGOVKTKbmghRky/k+fhER2VM+tPhFRKQdFX4RkTyTE4XfzC42s3fMrNXMKjusu8XMlpvZUjM7K8bz+5vZc2a2LPK9Xwoy/sHMFka+VpvZwhiPW21mb0ceNy/ZOaJs73YzW9cu2zkxHjc5sg+Xm9nNach1l5m9Z2aLzOxPZlYa43Fp2V/7+/kt7KeR9YvM7NhUZWm3zaFm9qKZLYn8/l8f5TETzWxbu/f3O6nOFdnuPt+XgPbX6Hb7YaGZbTezGzo8Ji37y8weNLONZra43bK46lBS/hbdPeu/gMOA0cAcoLLd8sOBt4BiYASwAiiM8vz/Bm6O3L4Z+GGK8/4I+E6MdauBgWncd7cD39jPYwoj+24k0D2yTw9Pca4zgaLI7R/Gek/Ssb/i+fmBc4C/AgacCLyehvduCHBs5HZv4J9Rck0EnkrX71O870sQ+yvKe/oh4Quc0r6/gFOBY4HF7Zbttw4l628xJ1r87r7E3ZdGWXU+UOXuDe6+ClgOHB/jcQ9Fbj8EXJCSoIRbOsBngN+nahspcDyw3N1XunsjUEV4n6WMuz/r7s2Ru68BB6Zye/sRz89/PvAbD3sNKDWzIakM5e7r3f3NyO0dwBIgW2YbSvv+6uA0YIW7d3ZEgC5x95eBLR0Wx1OHkvK3mBOFfx8qgDXt7q8l+h9Gmbuvh/AfEzAohZlOATa4+7IY6x141szmm9nUFOZo78uRj9sPxvh4Ge9+TJWrCLcOo0nH/orn5w90H5nZcGAc8HqU1SeZ2Vtm9lczOyJNkfb3vgT9O3UpsRtfQewviK8OJWW/Zc0MXGb2PDA4yqpvu/sTsZ4WZVnKzl+NM+Nn2Xdrf4K7V5vZIOA5M3sv0jpISS7gPuAOwvvlDsLdUFd1fIkoz+3yfoxnf5nZt4Fm4JEYL5P0/RUtapRlHX/+tP6u7bFhsxJgBnCDu2/vsPpNwt0ZtZHjN48Do9IQa3/vS5D7qzvwKeCWKKuD2l/xSsp+y5rC7+6nd+Jpa4Gh7e4fCFRHedwGMxvi7usjHzc3piKjmRUBnwbG7+M1qiPfN5rZnwh/tOtSIYt335nZr4GnoqyKdz8mNZeZTQHOBU7zSAdnlNdI+v6KIp6fPyX7aH/MrBvhov+Iu8/suL79PwJ3f9rMfmFmA909pQOSxfG+BLK/Is4G3nT3DR1XBLW/IuKpQ0nZb7ne1fMkcKmZFZvZCML/ud+I8bgpkdtTgFifILrqdOA9d18bbaWZ9TKz3m23CR/gXBztscnSoV/132Jsby4wysxGRFpLlxLeZ6nMNRn4FvApd98Z4zHp2l/x/PxPAl+InK1yIrCt7WN7qkSOFz0ALHH3u2M8ZnDkcZjZ8YT/5jenOFc870va91c7MT91B7G/2omnDiXnbzHVR6/T8UW4YK0FGoANwKx2675N+Cj4UuDsdsvvJ3IGEDAAeAFYFvneP0U5pwPXdVhWDjwduT2S8FH6t4B3CHd5pHrf/RZ4G1gU+QUa0jFX5P45hM8aWZGmXMsJ92UujHz9Msj9Fe3nB65rez8JfwT/eWT927Q7uyyFmT5G+GP+onb76ZwOub4c2TdvET5IfnIackV9X4LeX5Ht9iRcyPu2W5b2/UX4H896oClSu66OVYdS8beoIRtERPJMrnf1iIhIByr8IiJ5RoVfRCTPqPCLiOQZFX4RkTyjwi+SIAuPirnKzPpH7veL3B8WdDaReKjwiyTI3dcQHurizsiiO4FpHtCAXyKJ0nn8Ip0QGSphPvAgcC0wzsOjJYpkvKwZq0ckk7h7k5l9E3gGOFNFX7KJunpEOu9swpfdjw06iEgiVPhFOsHMjgHOIDx71NfSPImISJeo8IskKDJ6432Ex7//ALgL+J9gU4nET4VfJHHXAh+4+3OR+78AxpjZxwPMJBI3ndUjIpJn1OIXEckzKvwiInlGhV9EJM+o8IuI5BkVfhGRPKPCLyKSZ1T4RUTyzP8B0oEQhvngQw0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "alpha = 0.1\n",
    "\n",
    "y = list(map(lambda a: a if a>=0 else a*0.1, x))\n",
    "plot_graph(y,\"ReLU(X)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffa3b46c",
   "metadata": {},
   "source": [
    "## Softmax\n",
    "\n",
    "Usually, the softmax function is used in the output layer of a neural network. Classification neural networks typically employ the softmax function. The neuron that has the highest value claims the input as a member of its class. Because it is a preferable method, the softmax activation function forces the output of the neural network to represent the probability that the input falls into each of the classes. Without the softmax, the neuron's outputs are simply numeric values, with the highest indicating the winning class.\n",
    "\n",
    "$$ softmax_i(x) = \\frac{e^{z_i}}{\\sum\\limits_{j}e^{z_j}} $$\n",
    "\n",
    "Here, the Z represents the values from the neurons of the output layer. The exponential acts as the non-linear function. Later these values are divided by the sum of exponential values in order to normalize and then convert them into probabilities.\n",
    "\n",
    "Note that, when the number of classes is two, it becomes the same as the sigmoid activation function. In other words, sigmoid is simply a variant of the Softmax function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2694b5e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.047384713149342166 0.0008678812948896053 0.9517474055557683\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "from math import exp\n",
    "arr = [6, 2, 9]\n",
    "p1 = exp(arr[0]) / (exp(arr[0])+ exp(arr[1])+exp(arr[2]) ) \n",
    "p2 = exp(arr[1]) / (exp(arr[0])+ exp(arr[1])+exp(arr[2]) ) \n",
    "p3 = exp(arr[2]) / (exp(arr[0])+ exp(arr[1])+exp(arr[2]) ) \n",
    "print(p1,p2,p3)\n",
    "print(p1+p2+p3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ecb05115",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 6 artists>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAALyElEQVR4nO3db4hlhXnH8e+vu4ZEYzDFabCu20khLAQpNQzpn4UUNJZtVmJe5IWCYlvLvGla0xbSlb4IfSe0hBRaWha1SYloixoaIqQuiSKCMZldN4m65g/pNtlquxOkJLYFu+3TF3tLN+PuzOw9Z+fuc+f7gWHuPXNnznMQv3s495xzU1VIkvr5iVkPIEmajgGXpKYMuCQ1ZcAlqSkDLklN7dzKlV155ZW1uLi4lauUpPYOHz78g6paWLt8SwO+uLjIysrKVq5SktpL8k9nW+4hFElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNbVhwJPcn+RkkufPWPYnSV5K8vUkn01yxQWdUpL0BpvZA/8UsG/NskPAtVX1c8C3gLtHnkuStIENA15VTwGvrln2eFWdmjz9MrDrAswmSVrHGFdi/ibwt+f6YZJlYBlg9+7dI6xOkv7f4oHHZj3Cphy/Z//of3PQm5hJ/gg4BTxwrtdU1cGqWqqqpYWFN1zKL0ma0tR74EnuAG4Cbig/l02SttxUAU+yD/hD4Feq6j/GHUmStBmbOY3wQeAZYE+SE0nuBP4cuBw4lORokr+6wHNKktbYcA+8qm49y+L7LsAskqTz4JWYktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNTXGBzpIc2s7f1iALn7ugUtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDW1YcCT3J/kZJLnz1j2k0kOJfn25PvbL+yYkqS1NrMH/ilg35plB4AvVtW7gC9OnkuSttCGAa+qp4BX1yy+Gfj05PGngQ+NO5YkaSPTHgN/R1W9AjD5/lPnemGS5SQrSVZWV1enXJ0kaa0L/iZmVR2sqqWqWlpYWLjQq5OkbWPagP9rkqsAJt9PjjeSJGkzpg3454A7Jo/vAP5+nHEkSZu1mdMIHwSeAfYkOZHkTuAe4MYk3wZunDyXJG2hDT+VvqpuPcePbhh5FknSefBKTElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqyoBLUlODAp7k95K8kOT5JA8mefNYg0mS1jd1wJNcDfwusFRV1wI7gFvGGkyStL6hh1B2Am9JshO4FHh5+EiSpM3YOe0vVtU/J/lT4HvAfwKPV9Xja1+XZBlYBti9e/e0q5tLiwcem/UIm3L8nv2zHkHSWQw5hPJ24GbgncBPA5cluW3t66rqYFUtVdXSwsLC9JNKkn7MkEMo7wf+sapWq+q/gEeBXx5nLEnSRoYE/HvALya5NEmAG4Bj44wlSdrI1AGvqmeBh4EjwDcmf+vgSHNJkjYw9ZuYAFX1ceDjI80iSToPXokpSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0NCniSK5I8nOSlJMeS/NJYg0mS1rdz4O//GfCFqvpwkjcBl44wkyRpE6YOeJK3Ae8Dfh2gql4HXh9nLEnSRoYcQvlZYBX46yTPJbk3yWVrX5RkOclKkpXV1dUBq5MknWlIwHcC7wH+sqquA/4dOLD2RVV1sKqWqmppYWFhwOokSWcaEvATwImqenby/GFOB12StAWmDnhV/Qvw/SR7JotuAF4cZSpJ0oaGnoXyO8ADkzNQvgv8xvCRJEmbMSjgVXUUWBpnFEnS+fBKTElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLU1NC7EW6ZxQOPzXqETTl+z/5ZjyBpm3APXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqanBAU+yI8lzST4/xkCSpM0ZYw/8LuDYCH9HknQeBgU8yS5gP3DvOONIkjZr6Ac6fBL4GHD5uV6QZBlYBti9e/fA1eli5oduSFtr6j3wJDcBJ6vq8Hqvq6qDVbVUVUsLCwvTrk6StMaQQyh7gQ8mOQ48BFyf5DOjTCVJ2tDUAa+qu6tqV1UtArcAX6qq20abTJK0Ls8Dl6SmRvlU+qp6EnhyjL8lSdoc98AlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqapR7oUjqwQ/dmC/ugUtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTBlySmjLgktSUAZekpgy4JDU1dcCTXJPkiSTHkryQ5K4xB5MkrW/I7WRPAX9QVUeSXA4cTnKoql4caTZJ0jqm3gOvqleq6sjk8Y+AY8DVYw0mSVrfKMfAkywC1wHPnuVny0lWkqysrq6OsTpJEiMEPMlbgUeAj1bVD9f+vKoOVtVSVS0tLCwMXZ0kaWJQwJNcwul4P1BVj44zkiRpM4achRLgPuBYVX1ivJEkSZsxZA98L3A7cH2So5OvD4w0lyRpA1OfRlhVTwMZcRZJ0nnwSkxJasqAS1JTBlySmjLgktSUAZekpgy4JDVlwCWpKQMuSU0ZcElqyoBLUlMGXJKaMuCS1JQBl6SmDLgkNWXAJakpAy5JTRlwSWrKgEtSUwZckpoy4JLUlAGXpKYMuCQ1ZcAlqSkDLklNGXBJasqAS1JTgwKeZF+Sbyb5TpIDYw0lSdrY1AFPsgP4C+DXgHcDtyZ591iDSZLWN2QP/L3Ad6rqu1X1OvAQcPM4Y0mSNpKqmu4Xkw8D+6rqtybPbwd+oao+suZ1y8Dy5Oke4JvTjzu6K4EfzHqIkc3bNs3b9sD8bdO8bQ9cfNv0M1W1sHbhzgF/MGdZ9oZ/DarqIHBwwHoumCQrVbU06znGNG/bNG/bA/O3TfO2PdBnm4YcQjkBXHPG813Ay8PGkSRt1pCAfxV4V5J3JnkTcAvwuXHGkiRtZOpDKFV1KslHgH8AdgD3V9ULo022NS7KQzsDzds2zdv2wPxt07xtDzTZpqnfxJQkzZZXYkpSUwZckpratgGft9sAJLk/yckkz896ljEkuSbJE0mOJXkhyV2znmmIJG9O8pUkX5tszx/PeqYxJNmR5Lkkn5/1LGNIcjzJN5IcTbIy63k2si2PgU9uA/At4EZOnw75VeDWqnpxpoMNkOR9wGvA31TVtbOeZ6gkVwFXVdWRJJcDh4EPdf1vlCTAZVX1WpJLgKeBu6rqyzMebZAkvw8sAW+rqptmPc9QSY4DS1V1MV3Ec07bdQ987m4DUFVPAa/Oeo6xVNUrVXVk8vhHwDHg6tlONb067bXJ00smX633npLsAvYD9856lu1quwb8auD7Zzw/QeM4zLski8B1wLMzHmWQyeGGo8BJ4FBVtd4e4JPAx4D/mfEcYyrg8SSHJ7cBuaht14Bv6jYAmr0kbwUeAT5aVT+c9TxDVNV/V9XPc/qq5fcmaXuoK8lNwMmqOjzrWUa2t6rew+m7rP725NDkRWu7BtzbADQwOVb8CPBAVT0663nGUlX/BjwJ7JvtJIPsBT44OWb8EHB9ks/MdqThqurlyfeTwGc5fbj1orVdA+5tAC5ykzf97gOOVdUnZj3PUEkWklwxefwW4P3ASzMdaoCquruqdlXVIqf///lSVd0247EGSXLZ5A1zklwG/CpwUZ/VtS0DXlWngP+7DcAx4O8a3gbgxyR5EHgG2JPkRJI7Zz3TQHuB2zm9Z3d08vWBWQ81wFXAE0m+zukdiENVNRen3s2RdwBPJ/ka8BXgsar6woxnWte2PI1QkubBttwDl6R5YMAlqSkDLklNGXBJasqAS1JTBlySmjLgktTU/wKJRJakO3zGaQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = [5, 8, 4, 11, 3, 12]\n",
    "plt.bar(range(len(data)), data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "8f86350a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6.57181045e-04 1.31998342e-02 2.41763396e-04 2.65125756e-01\n",
      " 8.89397829e-05 7.20686525e-01]\n",
      "0.9999999999999999\n"
     ]
    }
   ],
   "source": [
    "def softmax(arr):\n",
    "    e = np.exp(arr)\n",
    "    return e / e.sum()\n",
    "res = softmax(data)\n",
    "print(res)\n",
    "print(np.sum(res))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9471f612",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 6 artists>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAO8ElEQVR4nO3db4xdeV3H8ffHKY2yiqgdhbSVNlok1QDiWDSooHG1y652iSR2UYgKaWqsQoxKeSCJ4QkbEkOU4qTBhhgNDcnyp2EHClH+iIDMFMtKdymZ1JWOxXQABReJpcvXB3Mhl7t35p7p3Nnb+e37lUz2nnN+vfO9mew7J2fuuZOqQpK09X3bpAeQJI2HQZekRhh0SWqEQZekRhh0SWrEtkl94x07dtSePXsm9e0laUs6d+7c56tqetixiQV9z549LCwsTOrbS9KWlOTfVzvmJRdJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJaoRBl6RGGHRJasTE7hSVpHHbc/zeSY/QyYOvvX1TntczdElqRKegJzmY5GKSxSTHhxz/4yTne1+fSvJwku8d/7iSpNWMDHqSKeAEcBuwH7gryf7+NVX1uqp6ZlU9E3gV8MGq+uImzCtJWkWXM/QDwGJVXaqqa8Bp4NAa6+8C3jKO4SRJ3XUJ+k7gct/2Um/fIyR5PHAQuGfjo0mS1qNL0DNkX62y9leAf1rtckuSI0kWkiwsLy93nVGS1EGXoC8Bu/u2dwFXVll7mDUut1TVyaqaqaqZ6emhf3BDknSDugR9HtiXZG+S7axE+8zgoiTfDTwXeOd4R5QkdTHyxqKqup7kGHAWmAJOVdWFJEd7x2d7S18AvLeqvrJp00qSVtXpTtGqmgPmBvbNDmy/GXjzuAaTJK2Pd4pKUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1wqBLUiMMuiQ1olPQkxxMcjHJYpLjq6x5XpLzSS4k+eB4x5QkjbJt1IIkU8AJ4FZgCZhPcqaq7u9b80TgjcDBqvpsku/fpHklSavocoZ+AFisqktVdQ04DRwaWPMi4G1V9VmAqro63jElSaN0CfpO4HLf9lJvX7+nAt+T5ANJziV5ybAnSnIkyUKSheXl5RubWJI0VJegZ8i+GtjeBvwEcDvwy8CfJnnqI/5R1cmqmqmqmenp6XUPK0la3chr6Kycke/u294FXBmy5vNV9RXgK0k+BDwD+MxYppQkjdTlDH0e2Jdkb5LtwGHgzMCadwI/m2RbkscDzwYeGO+okqS1jDxDr6rrSY4BZ4Ep4FRVXUhytHd8tqoeSPIe4D7g68CbqupTmzm4JOlbdbnkQlXNAXMD+2YHtl8HvG58o0mS1sM7RSWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhrRKehJDia5mGQxyfEhx5+X5EtJzve+Xj3+USVJa9k2akGSKeAEcCuwBMwnOVNV9w8s/cequmMTZpQkddDlDP0AsFhVl6rqGnAaOLS5Y0mS1qtL0HcCl/u2l3r7Bv10kk8meXeSHx32REmOJFlIsrC8vHwD40qSVtMl6Bmyrwa2PwE8paqeAfwl8I5hT1RVJ6tqpqpmpqen1zWoJGltXYK+BOzu294FXOlfUFVfrqqHeo/ngMcl2TG2KSVJI3UJ+jywL8neJNuBw8CZ/gVJnpQkvccHes/7hXEPK0la3ch3uVTV9STHgLPAFHCqqi4kOdo7Pgu8EPjdJNeBrwKHq2rwsowkaRONDDp88zLK3MC+2b7HbwDeMN7RJEnr4Z2iktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjegU9CQHk1xMspjk+BrrfjLJw0leOL4RJUldjAx6kingBHAbsB+4K8n+VdbdDZwd95CSpNG6nKEfABar6lJVXQNOA4eGrPt94B7g6hjnkyR11CXoO4HLfdtLvX3flGQn8AJgdq0nSnIkyUKSheXl5fXOKklaQ5egZ8i+Gth+PfDKqnp4rSeqqpNVNVNVM9PT0x1HlCR1sa3DmiVgd9/2LuDKwJoZ4HQSgB3A85Ncr6p3jGNISdJoXYI+D+xLshf4D+Aw8KL+BVW19xuPk7wZeJcxl6RH18igV9X1JMdYeffKFHCqqi4kOdo7vuZ1c0nSo6PLGTpVNQfMDewbGvKq+q2NjyVJWi/vFJWkRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRhh0SWqEQZekRnQKepKDSS4mWUxyfMjxQ0nuS3I+yUKSnxn/qJKktYz8I9FJpoATwK3AEjCf5ExV3d+37O+BM1VVSZ4OvBV42mYMLEkarssZ+gFgsaouVdU14DRwqH9BVT1UVdXbvAUoJEmPqi5B3wlc7tte6u37FklekOTTwL3A74xnPElSV12CniH7HnEGXlVvr6qnAXcCrxn6RMmR3jX2heXl5XUNKklaW5egLwG7+7Z3AVdWW1xVHwJ+KMmOIcdOVtVMVc1MT0+ve1hJ0uq6BH0e2Jdkb5LtwGHgTP+CJD+cJL3HzwK2A18Y97CSpNWNfJdLVV1Pcgw4C0wBp6rqQpKjveOzwK8BL0nyNeCrwK/3/ZJUkvQoGBl0gKqaA+YG9s32Pb4buHu8o0mS1sM7RSWpEQZdkhph0CWpEQZdkhph0CWpEZ3e5SIJ9hy/d9IjdPLga2+f9AiaEM/QJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGmHQJakRBl2SGtEp6EkOJrmYZDHJ8SHHfyPJfb2vjyR5xvhHlSStZWTQk0wBJ4DbgP3AXUn2Dyz7N+C5VfV04DXAyXEPKklaW5cz9APAYlVdqqprwGngUP+CqvpIVf1Xb/NjwK7xjilJGqVL0HcCl/u2l3r7VvNS4N3DDiQ5kmQhycLy8nL3KSVJI3UJeobsq6ELk59nJeivHHa8qk5W1UxVzUxPT3efUpI0Upe/KboE7O7b3gVcGVyU5OnAm4DbquoL4xlPktRVlzP0eWBfkr1JtgOHgTP9C5L8IPA24MVV9ZnxjylJGmXkGXpVXU9yDDgLTAGnqupCkqO947PAq4HvA96YBOB6Vc1s3tiSpEFdLrlQVXPA3MC+2b7HLwNeNt7RJEnr4Z2iktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjTDoktQIgy5JjegU9CQHk1xMspjk+JDjT0vy0ST/l+SPxj+mJGmUbaMWJJkCTgC3AkvAfJIzVXV/37IvAn8A3LkZQ0qSRutyhn4AWKyqS1V1DTgNHOpfUFVXq2oe+NomzChJ6qBL0HcCl/u2l3r71i3JkSQLSRaWl5dv5CkkSavoEvQM2Vc38s2q6mRVzVTVzPT09I08hSRpFV2CvgTs7tveBVzZnHEkSTeqS9DngX1J9ibZDhwGzmzuWJKk9Rr5Lpequp7kGHAWmAJOVdWFJEd7x2eTPAlYAJ4AfD3JK4D9VfXlzRtdktRvZNABqmoOmBvYN9v3+D9ZuRQjSZoQ7xSVpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEZ0+pui2nx7jt876RE6efC1t096BEmr6HSGnuRgkotJFpMcH3I8Sf6id/y+JM8a/6iSpLWMDHqSKeAEcBuwH7gryf6BZbcB+3pfR4C/GvOckqQRupyhHwAWq+pSVV0DTgOHBtYcAv6mVnwMeGKSJ495VknSGrpcQ98JXO7bXgKe3WHNTuBz/YuSHGHlDB7goSQX1zXt5toBfH7SQ4zZ2F9T7h7ns62bP6MO/BmN3c32M3rKage6BD1D9tUNrKGqTgInO3zPR12ShaqamfQc49Taa2rt9UB7r6m11wNb6zV1ueSyBOzu294FXLmBNZKkTdQl6PPAviR7k2wHDgNnBtacAV7Se7fLTwFfqqrPDT6RJGnzjLzkUlXXkxwDzgJTwKmqupDkaO/4LDAHPB9YBP4X+O3NG3nT3JSXgjaotdfU2uuB9l5Ta68HttBrStUjLnVLkrYgb/2XpEYYdElqhEFn9EcbbDVJTiW5muRTk55lHJLsTvL+JA8kuZDk5ZOeaSOSfHuSjyf5ZO/1/NmkZxqXJFNJ/iXJuyY9y0YleTDJvyY5n2Rh0vN08Zi/ht77aIPPALey8vbLeeCuqrp/ooNtQJKfAx5i5e7dH5v0PBvVu+v4yVX1iSTfBZwD7tyqP6MkAW6pqoeSPA74MPDy3l3WW1qSPwRmgCdU1R2TnmcjkjwIzFTVlrlRyjP0bh9tsKVU1YeAL056jnGpqs9V1Sd6j/8HeICVO5G3pN5HZDzU23xc72vLn1kl2QXcDrxp0rM8Vhn01T+2QDehJHuAHwf+ecKjbEjv0sR54Crwvqra0q+n5/XAnwBfn/Ac41LAe5Oc631syU3PoHf82AJNXpLvBO4BXlFVX570PBtRVQ9X1TNZuav6QJItfWksyR3A1ao6N+lZxug5VfUsVj5N9vd6lzJvagbdjy3YEnrXmu8B/q6q3jbpecalqv4b+ABwcLKTbNhzgF/tXXc+DfxCkr+d7EgbU1VXev+9CrydlcuzNzWD3u2jDTRBvV8i/jXwQFX9+aTn2agk00me2Hv8HcAvAp+e6FAbVFWvqqpdVbWHlf+H/qGqfnPCY92wJLf0fgFPkluAXwJu+neNPeaDXlXXgW98tMEDwFur6sJkp9qYJG8BPgr8SJKlJC+d9Ewb9Bzgxayc9Z3vfT1/0kNtwJOB9ye5j5UTivdV1ZZ/m19jfgD4cJJPAh8H7q2q90x4ppEe829blKRWPObP0CWpFQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEf8P3NQFUYsoJ1QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(res)), res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d077525",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf24_py38_gpu",
   "language": "python",
   "name": "tf24_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}