Devanagri_Characters_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Devanagri Script Character Recognition using Convolutional Neural Networks:\n",
    "\n",
    "Devanagari is an Indic script. It consists of 47 primary alphabets, 14 vowels, and 33 consonants, and 10 digits.\n",
    "\n",
    "__About the data (Extracted Images):__\n",
    "\n",
    "The dataset consists of 92000 rows (sample greyscale images) of size (32 x 32), and 1025 columns. Each row contains the pixel data (\"pixel0000\" to \"pixel1023\"), in greyscale values (0 to 255). The column _character_ represents the Devanagari character name corresponding to each image.\n",
    "\n",
    "\n",
    "Refer for dataset:\n",
    "\n",
    "https://www.kaggle.com/rishianand/devanagari-character-set/data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/opt/conda/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/opt/conda/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/opt/conda/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/opt/conda/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/opt/conda/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import math\n",
    "import tensorflow_model_optimization as tfmot\n",
    "from tensorflow_model_optimization.sparsity import keras as sparsity\n",
    "# from tensorflow.keras import datasets, layers, models\n",
    "import matplotlib.pyplot as plt\n",
    "from tensorflow.keras.layers import AveragePooling2D, Conv2D, MaxPooling2D, ReLU\n",
    "from tensorflow.keras import models, layers, datasets\n",
    "from tensorflow.keras.layers import Dense, Flatten, Reshape, Input, InputLayer\n",
    "from tensorflow.keras.models import Sequential, Model\n",
    "from tensorflow.keras.initializers import RandomNormal\n",
    "# import math\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data preprocessing and cleadning:\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 32, 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read-in CSV file containing 92,000 images-\n",
    "data = pd.read_csv(\"devanagri_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(92000, 1025)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get shape/dimension of dataset-\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Target attribute is 'character'-\n",
    "'character' in data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split dataset into features (X) and target (y)-\n",
    "X = data.drop('character', axis = 1)\n",
    "y = data['character']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to numpy arrays-\n",
    "X = X.values\n",
    "y = y.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((92000, 1024), (92000,))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "character_12_thaa            2000\n",
       "digit_7                      2000\n",
       "character_01_ka              2000\n",
       "character_23_ba              2000\n",
       "character_04_gha             2000\n",
       "character_27_ra              2000\n",
       "character_25_ma              2000\n",
       "character_17_tha             2000\n",
       "digit_8                      2000\n",
       "character_13_daa             2000\n",
       "character_32_patalosaw       2000\n",
       "character_29_waw             2000\n",
       "digit_3                      2000\n",
       "digit_1                      2000\n",
       "character_10_yna             2000\n",
       "digit_6                      2000\n",
       "character_21_pa              2000\n",
       "character_35_tra             2000\n",
       "character_02_kha             2000\n",
       "character_33_ha              2000\n",
       "character_36_gya             2000\n",
       "character_34_chhya           2000\n",
       "character_24_bha             2000\n",
       "character_09_jha             2000\n",
       "character_19_dha             2000\n",
       "character_07_chha            2000\n",
       "character_08_ja              2000\n",
       "character_31_petchiryakha    2000\n",
       "digit_9                      2000\n",
       "character_06_cha             2000\n",
       "character_16_tabala          2000\n",
       "character_15_adna            2000\n",
       "character_22_pha             2000\n",
       "character_28_la              2000\n",
       "digit_0                      2000\n",
       "digit_2                      2000\n",
       "character_05_kna             2000\n",
       "character_11_taamatar        2000\n",
       "character_18_da              2000\n",
       "character_14_dhaa            2000\n",
       "digit_4                      2000\n",
       "character_03_ga              2000\n",
       "character_30_motosaw         2000\n",
       "character_26_yaw             2000\n",
       "digit_5                      2000\n",
       "character_20_na              2000\n",
       "Name: character, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check distribution of target attribute-\n",
    "data['character'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAATJElEQVR4nO3df5BddXnH8feTzc1uyC+Im2AM+WEwVAJqyOwElNYRURup5Uc7ooxjY4e6tsKMFlvLUKdgh3Zop4B0OsUGSY0VUVqlokNVTGkZBxuzYMgPkyJgGkJCQggJmyy72R9P/7gnzhLPc/bu/bnh+3nNZPbu97nnnidn73PPved7v9+vuTsi8to3qdUJiEhzqNhFEqFiF0mEil0kESp2kUSo2EUSMbmWjc1sFXAH0AZ8yd1vKbr/FGv3DqbVsksRKdDPUY75gOXFrNp+djNrA54E3gvsBjYCV7n7z6JtZtpsP98urmp/IjK2Db6el/1gbrHX8jZ+JfCUuz/j7seArwOX1fB4ItJAtRT7fODZUb/vztpEZAKq5TN73luFX/lMYGbdQDdAB6fUsDsRqUUtZ/bdwIJRv58B7DnxTu6+xt273L2rRHsNuxORWtRS7BuBpWb2RjObAnwYeKA+aYlIvVX9Nt7dh8zsWuD7lLve1rr7trplJiJ1VVM/u7s/CDxYp1xEpIH0DTqRRKjYRRKhYhdJhIpdJBEqdpFE1HQ1vlkmL1mc2/7KmzrDbXxS7liAciwOYSPxwKDJ/cO57W1HB8NtJvX2h7HBuTPCWGl/bxjjwME4Nqktv91H4m2G45gPDRVsl388ABjJf0wv2FdRjl60ryKaUPWXdGYXSYSKXSQRKnaRRKjYRRKhYhdJxElxNf7t/74jt/1jp/aE2wTXpMdUdM13ILiw2ztSCrd5fnhmGFtSiq+qbxp4Qxhbf2hZGGuflH/1fMTj1/VDg1PD2IH+OP/DAx1x7Gj+Y/YfKRjmfDR+Ok7bGf9FS0fiK+7th/JjHQfjXoZTtu0NY8MvHAhjfuxYGKtKnXsSdGYXSYSKXSQRKnaRRKjYRRKhYhdJhIpdJBEnRdfb5zrzu95gelPzqM5AQSxeCuus0uEwduX0H9eQz8lpuGCQzMivzmD+S4Oe35l6xOPBS189/JYw9qUd7whjpR/F3ZSlo3GOUa9o5z/V9++sM7tIIlTsIolQsYskQsUukggVu0giVOwiiTCvYWSNme0EeikPFhty966i+8+02X6+XTzu/fzG5vx53IpGvXVYwURzBfoKjsf8tvxVaNssfs3sG6luJNQpk6aEsaJuqCcH849Vm8X/r7NKcRdgtaIcBzwebdZX0B1WKjjGsybFo/ZOZr/5huXj3maDr+dlP5j75K9HP/tF7h6P+xORCUFv40USUWuxO/ADM3vMzLrrkZCINEatb+MvdPc9ZjYXeMjMdrj7I6PvkL0IdAN0kP+ZV0Qar6Yzu7vvyX7uB+4HVubcZ427d7l7V4mCKYlEpKGqLnYzm2ZmM47fBt4HbK1XYiJSX7W8jT8duN/KXVyTga+5+/fqktUJfnzJmbnt//nmC8NthqcWTDlZ0NvYe0a83cOfuy23va1gPam3f+G6MNY3P+5Ce/pDXwxj/3jojWHsO90X5bYfmxVPirnmzi+EsWq75c59dHVu+8wH4pGKHYfi6T6H2+NjfHhJ/Dfr78z/Y894czzZ57LOfWHs8/O/G8aOelxOB4fjj7CPv7I4jNVT1cXu7s8Ab6tjLiLSQOp6E0mEil0kESp2kUSo2EUSoWIXScRJMeHk0O7ncttLQTtA3NFU7Pm/ensYi0ZXXfvc+eE2C9bEXz147upzK09slDmTXw5jk7f9Ir+9Pf5C0/PDRRNfVp7XaP0v5a8Dt/Cr/xNvVOUIzGlFIxyD0XJWKugmOyXuJrvmzD+Md9Ufj+iz/njiUTvSF0TiLsBq6MwukggVu0giVOwiiVCxiyRCxS6SiJPianzdTYoHTrSfc2jcD/fgo+eFsaW9G8JY/+zq5/8LRXO1DcRXg58dfF38eB0vVpXG9DlHq9quKkVX8YPln3ygYNBNwbGi56V4V/FWE4LO7CKJULGLJELFLpIIFbtIIlTsIolQsYskIsmuN5sUD5xYeOr4u95mPl3wmlnQLbRgfdzF88lVF4Sxh7+zIowt6v1JbrtNiZeTeqr/9DDGjOq63hadlt9FNViwjFPUTSb1oTO7SCJU7CKJULGLJELFLpIIFbtIIlTsIokYs+vNzNYCHwD2u/u5Wdts4BvAYmAncKW7x8OBJhibHP+3z575/Lgfb/qe6rqM2h55IoztfP+pYWxR72NhzIfiedAiPz20IA7O+dm4Hw9gZqk/t/1gQbenx6thSR1Ucmb/MrDqhLbrgfXuvhRYn/0uIhPYmMWerbd+4ip4lwHrstvrgMvrnJeI1Fm1n9lPd/e9ANnPufVLSUQaoeFflzWzbqAboIN4Pm4Raaxqz+z7zGweQPZzf3RHd1/j7l3u3lUiXqhARBqr2mJ/AFid3V4NfLs+6YhIo1TS9XYv8C6g08x2AzcCtwD3mdnVwC7gg41M8jVrpGDSwwPVjTaL+HC8rydfmBNvuLS6/b3YHywpNRIvXSWNNWaxu/tVQejiOuciIg2kb9CJJELFLpIIFbtIIlTsIolQsYskIskJJ0eODYaxjQcWxRu+/qe5zQfPjteOm16KJ3os4oPHqtoufsB44stXXppa330B+49Mz22fq6FtLaMzu0giVOwiiVCxiyRCxS6SCBW7SCJU7CKJSLLrrWhmw107CtY9Oze/+Zbf+3K4yR+f9aEwNnKkFMbOvvWFMDb81C/CWDWsL+46rFbvkfzuPE1p1Do6s4skQsUukggVu0giVOwiiVCxiyQi0avx8aCQhf8RX6k//Duv5LZfGky3BnDpxf8cxoYLegXO2X9tGFt0Y32vxk8+Gi/JVLU9HfntBcdeGktndpFEqNhFEqFiF0mEil0kESp2kUSo2EUSUcnyT2uBDwD73f3crO0m4OPA8dEaN7j7g41Kspmm/ve2MLb8O5/Kbd/023eE28yaFM/v1mbxa+3grObN1daIrrdpuxvQnSc1qeTM/mVgVU777e6+PPv3mih0kdeyMYvd3R8BDjYhFxFpoFo+s19rZpvNbK2ZnVa3jESkIaot9juBM4HlwF7g1uiOZtZtZj1m1jPIQJW7E5FaVVXs7r7P3YfdfQS4C1hZcN817t7l7l0l2qvNU0RqVFWxm9m8Ub9eAWytTzoi0iiVdL3dC7wL6DSz3cCNwLvMbDngwE7gEw3MsalG+vrC2LJb9ua2339RvGTUx2buD2NPDx4JYwu+PxzG6q0tXg2r0IDHG3Zu0Ue2iWbMYnf3q3Ka725ALiLSQPoGnUgiVOwiiVCxiyRCxS6SCBW7SCLSnHCyiMWjtZ793QW57R+d8e1wm8MjcRfUpV/8bBhb8MOeMFbvKRtHqnwWfL9vVhhr3/ZsbnvzOhTlRDqziyRCxS6SCBW7SCJU7CKJULGLJELFLpIIdb2doK2zM4x95Pcfyt+mYOLIdz++Oowt/IctYWxk8FgYq8qktjDUt7i6YW+373xvGGt/8bmqHlMaR2d2kUSo2EUSoWIXSYSKXSQRKnaRROhq/An6Vi4OY9fN/l5u+0vD8WCXWXfMCGMjvU9WnFet2t60OIzdcdE9VT3m84dmhrGFw7uqekxpHJ3ZRRKhYhdJhIpdJBEqdpFEqNhFEqFiF0lEJcs/LQC+ArweGAHWuPsdZjYb+AawmPISUFe6+0uNS7U5XnhrKYyVLH8wyS0H3hFu074h7l4bqTytillpSm77jmvmhNtcOi1e8qpIW1sj/gfSKJWc2YeAz7j72cAFwDVmtgy4Hljv7kuB9dnvIjJBjVns7r7X3R/PbvcC24H5wGXAuuxu64DLG5WkiNRuXJ/ZzWwxcB6wATjd3fdC+QUBmFvv5ESkfioudjObDnwT+LS7vzyO7brNrMfMegbRMr4irVJRsZtZiXKh3+Pu38qa95nZvCw+D8hdiNzd17h7l7t3lWivR84iUoUxi93MjPJ67Nvd/bZRoQeA43MurQbiZVFEpOUqGfV2IfBRYIuZbcrabgBuAe4zs6uBXcAHG5Nic/XPHX930g92vTmMzT3681rSyVewRFXvFSty2x+6/O8KHnB6VWkcffGUOOj1XqRKajVmsbv7j4Do2XVxfdMRkUbRN+hEEqFiF0mEil0kESp2kUSo2EUSkeaEkwVdVz5l/F1GH17yWBh7eMX5cRo/eybOY3g4jA2945wwdt3NX8ttP7NUXfdakckH03z6nKx0ZhdJhIpdJBEqdpFEqNhFEqFiF0mEil0kEWn2nRSMyOrcGL/+DV6e3x32Z6+LR7b92r17w9hfP/n+MHaod2oYu70rv3sN4LdO6c9t33bslXCbew+tDGM3z90SxoZmxd2DMvHozC6SCBW7SCJU7CKJULGLJELFLpKINK/GF+j8brxc00f+4H257fctWR9uc/m0I3HsvH+tPLEKHRg+mtv+oS/+abhN37L8K/gAN78nvhovJxed2UUSoWIXSYSKXSQRKnaRRKjYRRKhYhdJxJhdb2a2APgK8HpgBFjj7neY2U3Ax4EXsrve4O4PNirRZhk+8GIYO/Lxpbnt59z8kXCbe1asDWNvKsUDcoaJYz/sOz2M/eWdn8xtX3Dn4+E2O/7+LWGsiA3oXHEyqaSffQj4jLs/bmYzgMfM7KEsdru7Fy0iJiITRCVrve0F9ma3e81sOzC/0YmJSH2N632YmS0GzgM2ZE3XmtlmM1trZqfVOTcRqaOKi93MpgPfBD7t7i8DdwJnAsspn/lvDbbrNrMeM+sZZKAOKYtINSoqdjMrUS70e9z9WwDuvs/dh919BLgLyJ3uxN3XuHuXu3eVaK9X3iIyTmMWu5kZcDew3d1vG9U+b9TdrgC21j89EamXSq7GXwh8FNhiZpuythuAq8xsOeDATuATDclwAhnenj/X3MKPzQi3+exbusNY76J4nrlJQ3HX28zth8PYvB0/yW0fGRqK93W0LYwV8akFc9BFS2wVzP8njVXJ1fgfAXl/uZO+T10kJfpWhEgiVOwiiVCxiyRCxS6SCBW7SCI04WQdjPT2hjF79IkwNvPRKvdX3Wah6buqe80/56zdYWyoLb87zwu6AKWxdGYXSYSKXSQRKnaRRKjYRRKhYhdJhIpdJBHqehPmbuwLY78YjNeqGxiOnz7VjaOTRtKZXSQRKnaRRKjYRRKhYhdJhIpdJBEqdpFEqOtNaNu4PYy9+8HrwljHvvjps3A4HhEnraEzu0giVOwiiVCxiyRCxS6SCBW7SCLGvBpvZh3AI0B7dv9/c/cbzWw28A1gMeXln65095cal6o0ig/Eq+ue9Uf5y0nJyaeSM/sA8G53fxvl5ZlXmdkFwPXAendfCqzPfheRCWrMYvey4+McS9k/By4D1mXt64DLG5KhiNRFpeuzt2UruO4HHnL3DcDp7r4XIPs5t3FpikitKip2dx929+XAGcBKMzu30h2YWbeZ9ZhZzyDxZ0MRaaxxXY1390PAfwGrgH1mNg8g+7k/2GaNu3e5e1eJ9hrTFZFqjVnsZjbHzE7Nbk8F3gPsAB4AVmd3Ww18u1FJikjtKhkIMw9YZ2ZtlF8c7nP375rZj4H7zOxqYBfwwQbmKSI1GrPY3X0zcF5O+4vAxY1ISkTqT9+gE0mEil0kESp2kUSo2EUSoWIXSYS5e/N2ZvYC8H/Zr53AgabtPKY8Xk15vNrJlscid5+TF2hqsb9qx2Y97t7Vkp0rD+WRYB56Gy+SCBW7SCJaWexrWrjv0ZTHqymPV3vN5NGyz+wi0lx6Gy+SiJYUu5mtMrP/NbOnzKxlc9eZ2U4z22Jmm8ysp4n7XWtm+81s66i22Wb2kJn9PPt5WovyuMnMnsuOySYzu6QJeSwws4fNbLuZbTOzT2XtTT0mBXk09ZiYWYeZ/cTMnsjy+HzWXtvxcPem/gPagKeBJcAU4AlgWbPzyHLZCXS2YL/vBFYAW0e1/S1wfXb7euBvWpTHTcCfNPl4zANWZLdnAE8Cy5p9TAryaOoxAQyYnt0uARuAC2o9Hq04s68EnnL3Z9z9GPB1ypNXJsPdHwEOntDc9Ak8gzyazt33uvvj2e1eYDswnyYfk4I8msrL6j7JayuKfT7w7Kjfd9OCA5px4Adm9piZdbcoh+Mm0gSe15rZ5uxtfsM/ToxmZospz5/Q0klNT8gDmnxMGjHJayuK3XLaWtUlcKG7rwDeD1xjZu9sUR4TyZ3AmZTXCNgL3NqsHZvZdOCbwKfd/eVm7beCPJp+TLyGSV4jrSj23cCCUb+fAexpQR64+57s537gfsofMVqlogk8G83d92VPtBHgLpp0TMysRLnA7nH3b2XNTT8meXm06phk+x73JK+RVhT7RmCpmb3RzKYAH6Y8eWVTmdk0M5tx/DbwPmBr8VYNNSEm8Dz+ZMpcQROOiZkZcDew3d1vGxVq6jGJ8mj2MWnYJK/NusJ4wtXGSyhf6Xwa+PMW5bCEck/AE8C2ZuYB3Ev57eAg5Xc6VwOvo7yM1s+zn7NblMe/AFuAzdmTa14T8vh1yh/lNgObsn+XNPuYFOTR1GMCvBX4aba/rcBfZO01HQ99g04kEfoGnUgiVOwiiVCxiyRCxS6SCBW7SCJU7CKJULGLJELFLpKI/wduqi9FGwz/LQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize a character from dataset-\n",
    "plt.imshow(X[6000, :].reshape(32, 32))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(92000, 32, 32, 1)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reshape data for CNN-\n",
    "X_exp = X.reshape(X.shape[0], 32, 32, 1)\n",
    "X_exp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Label encode target attribute-\n",
    "\n",
    "# Initialize a label encoder-\n",
    "le = LabelEncoder()\n",
    "\n",
    "# Train label encoder on target attribute-\n",
    "y_transformed = le.fit_transform(y)\n",
    "\n",
    "# To get back original target-\n",
    "# y_orig = le.inverse_transform(y_transformed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of classes for target attribute = 46\n"
     ]
    }
   ],
   "source": [
    "# number of classes for target attribute-\n",
    "print(\"number of classes for target attribute = {0}\".format(len(set(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 60\n",
    "num_classes = len(set(y))\n",
    "num_epochs = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split features and target attribute into training and testing sets-\n",
    "# X_train, X_test, y_train, y_test = train_test_split(data.drop('character',axis=1), y, test_size=0.3)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_exp, y_transformed, test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert datasets to floating point types-\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "\n",
    "# Normalize the training and testing datasets-\n",
    "X_train /= 255.0\n",
    "X_test /= 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert class vectors/target to binary class matrices or one-hot encoded values-\n",
    "y_train = tf.keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = tf.keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Shapes of training and testing sets are:\n",
      "X_train.shape = (64400, 32, 32, 1), y_train = (64400, 46)\n",
      "X_test.shape = (27600, 32, 32, 1), y_test = (27600, 46)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\nShapes of training and testing sets are:\")\n",
    "print(\"X_train.shape = {0}, y_train = {1}\".format(X_train.shape, y_train.shape))\n",
    "print(\"X_test.shape = {0}, y_test = {1}\\n\".format(X_test.shape, y_test.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare MNIST dataset for _GradientTape_ training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create training and testing datasets-\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train))\n",
    "test_dataset = tf.data.Dataset.from_tensor_slices((X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = train_dataset.shuffle(buffer_size = 20000, reshuffle_each_iteration = True).batch(batch_size = batch_size, drop_remainder = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataset = test_dataset.batch(batch_size=batch_size, drop_remainder=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose an optimizer and loss function for training-\n",
    "loss_fn = tf.keras.losses.CategoricalCrossentropy()\n",
    "optimizer = tf.keras.optimizers.Adam(lr = 0.0012)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select metrics to measure the error & accuracy of model.\n",
    "# These metrics accumulate the values over epochs and then\n",
    "# print the overall result-\n",
    "train_loss = tf.keras.metrics.Mean(name = 'train_loss')\n",
    "train_accuracy = tf.keras.metrics.CategoricalAccuracy(name = 'train_accuracy')\n",
    "\n",
    "test_loss = tf.keras.metrics.Mean(name = 'test_loss')\n",
    "test_accuracy = tf.keras.metrics.CategoricalAccuracy(name = 'test_accuracy')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'end_step parameter' for this dataset =  107400\n"
     ]
    }
   ],
   "source": [
    "# The model is first trained without any pruning for 'num_epochs' epochs-\n",
    "epochs = num_epochs\n",
    "\n",
    "num_train_samples = X_train.shape[0]\n",
    "\n",
    "end_step = np.ceil(1.0 * num_train_samples / batch_size).astype(np.int32) * epochs\n",
    "\n",
    "print(\"'end_step parameter' for this dataset =  {0}\".format(end_step))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the parameters to be used for layer-wise pruning, NO PRUNING is done here:\n",
    "pruning_params_unpruned = {\n",
    "    'pruning_schedule': sparsity.ConstantSparsity(\n",
    "        target_sparsity=0.0, begin_step=0,\n",
    "        end_step = end_step, frequency=100\n",
    "    )\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "l = tf.keras.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pruned_nn(pruning_params_conv, pruning_params_fc):\n",
    "    \"\"\"\n",
    "    Function to define the architecture of a neural network model\n",
    "    following LeNet-5 architecture for MNIST dataset and using\n",
    "    provided parameter which are used to prune the model.\n",
    "    \n",
    "    \n",
    "    Input: 'pruning_params' Python 3 dictionary containing parameters which are used for pruning\n",
    "    Output: Returns designed and compiled neural network model\n",
    "    \"\"\"\n",
    "    \n",
    "    pruned_model = Sequential()\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 6, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'valid',\n",
    "            input_shape=(32, 32, 1)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "       \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 16, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'valid'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "   \n",
    "    pruned_model.add(Flatten())\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 120, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 84, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = num_classes, activation='softmax'\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "\n",
    "    # Compile pruned CNN-\n",
    "    pruned_model.compile(\n",
    "        loss=tf.keras.losses.categorical_crossentropy,\n",
    "        # optimizer='adam',\n",
    "        optimizer=tf.keras.optimizers.Adam(lr = 0.0012),\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    \n",
    "    \n",
    "    return pruned_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a pruning step callback to peg the pruning step to the optimizer's\n",
    "# step. Also add a callback to add pruning summaries to tensorboard\n",
    "callback = [\n",
    "             sparsity.UpdatePruningStep(),\n",
    "             # sparsity.PruningSummaries(log_dir = logdir, profile_batch=0),\n",
    "             tf.keras.callbacks.EarlyStopping(\n",
    "                 monitor='val_loss', patience = 3,\n",
    "                 min_delta=0.001\n",
    "             )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a CNN model-\n",
    "orig_model = pruned_nn(pruning_params_unpruned, pruning_params_unpruned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save weights for later-\n",
    "orig_model.save_weights(\"LeNet_5_Devanagri_Random_Weights.h5\", overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strip model of it's pruning parameters\n",
    "orig_model_stripped = sparsity.strip_pruning(orig_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_4 (Conv2D)            (None, 30, 30, 6)         60        \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 15, 15, 6)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 13, 13, 16)        880       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 6, 6, 16)          0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 576)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 120)               69240     \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 84)                10164     \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 46)                3910      \n",
      "=================================================================\n",
      "Total params: 84,254\n",
      "Trainable params: 84,254\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Get CNN summary-\n",
    "orig_model_stripped.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train defined CNN to get baseline performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1074/1074 [==============================] - 24s 22ms/step - loss: 0.9261 - accuracy: 0.7395 - val_loss: 0.4182 - val_accuracy: 0.8758\n",
      "Epoch 2/100\n",
      "1074/1074 [==============================] - 23s 21ms/step - loss: 0.3104 - accuracy: 0.9062 - val_loss: 0.3143 - val_accuracy: 0.9057\n",
      "Epoch 3/100\n",
      "1074/1074 [==============================] - 23s 21ms/step - loss: 0.2087 - accuracy: 0.9358 - val_loss: 0.2138 - val_accuracy: 0.9371\n",
      "Epoch 4/100\n",
      "1074/1074 [==============================] - 23s 21ms/step - loss: 0.1570 - accuracy: 0.9503 - val_loss: 0.1944 - val_accuracy: 0.9429\n",
      "Epoch 5/100\n",
      "1074/1074 [==============================] - 23s 22ms/step - loss: 0.1234 - accuracy: 0.9613 - val_loss: 0.2037 - val_accuracy: 0.9404\n",
      "Epoch 6/100\n",
      "1074/1074 [==============================] - 23s 22ms/step - loss: 0.1044 - accuracy: 0.9661 - val_loss: 0.1692 - val_accuracy: 0.9521\n",
      "Epoch 7/100\n",
      "1074/1074 [==============================] - 25s 23ms/step - loss: 0.0845 - accuracy: 0.9714 - val_loss: 0.1762 - val_accuracy: 0.9512\n",
      "Epoch 8/100\n",
      "1074/1074 [==============================] - 25s 23ms/step - loss: 0.0726 - accuracy: 0.9760 - val_loss: 0.1849 - val_accuracy: 0.9486\n",
      "Epoch 9/100\n",
      "1074/1074 [==============================] - 24s 22ms/step - loss: 0.0628 - accuracy: 0.9788 - val_loss: 0.1751 - val_accuracy: 0.9549\n"
     ]
    }
   ],
   "source": [
    "# Sanity check: Whether defined CNN works as intended-\n",
    "\n",
    "history = orig_model.fit(\n",
    "    x = X_train, y = y_train,\n",
    "    epochs=num_epochs, batch_size=batch_size,\n",
    "    verbose=1, shuffle=True,\n",
    "    callbacks=callback,\n",
    "    validation_data=(X_test, y_test)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Trained LeNet-5 CNN model metrics on validation set:\n",
      "val_loss = 0.17512 and val_accuracy = 95.48913%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Trained CNN performance on validation data set-\n",
    "model_val_loss, model_val_acc = orig_model.evaluate(X_test, y_test, verbose=0)\n",
    "\n",
    "print(\"\\nTrained LeNet-5 CNN model metrics on validation set:\")\n",
    "print(\"val_loss = {0:.5f} and val_accuracy = {1:.5f}%\\n\".format(\n",
    "    model_val_loss, model_val_acc * 100\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question:\n",
    "\n",
    "Can the model's performance be further improved?\n",
    "\n",
    "To tackle this question, the following Convolutional Neural Networks are experimented with:\n",
    "\n",
    "1. Conv-2\n",
    "1. Conv-4\n",
    "1. Conv-6\n",
    "\n",
    "These CNNs are _VGG-16_ inspired. VGG-16 is a convolutional neural network model proposed by K. Simonyan and A. Zisserman from the University of Oxford in the paper “Very Deep Convolutional Networks for Large-Scale Image Recognition”.\n",
    "\n",
    "Originally, these CNNs were designed for _CIFAR-10_ dataset. But for our experiment, these CNNs will be used for the _Devanagri_ dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pruned_nn(pruning_params_conv, pruning_params_fc):\n",
    "    \"\"\"\n",
    "    Function to define the architecture of a neural network model\n",
    "    following Conv-2 architecture for CIFAR-10 dataset and using\n",
    "    provided parameter which are used to prune the model.\n",
    "    \n",
    "    Conv-2 architecture-\n",
    "    64, 64, pool  -- convolutions\n",
    "    256, 256, 46  -- fully connected layers\n",
    "    \n",
    "    Input: 'pruning_params' Python 3 dictionary containing parameters which are used for pruning\n",
    "    Output: Returns designed and compiled neural network model\n",
    "    \"\"\"\n",
    "    \n",
    "    pruned_model = Sequential()\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 64, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same',\n",
    "            input_shape=(32, 32, 1)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "        \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 64, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(Flatten())\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 256, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 256, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = num_classes, activation='softmax'\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "\n",
    "    # Compile pruned CNN-\n",
    "    pruned_model.compile(\n",
    "        loss=tf.keras.losses.categorical_crossentropy,\n",
    "        # optimizer='adam',\n",
    "        optimizer=tf.keras.optimizers.Adam(lr = 0.0012),\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    \n",
    "    \n",
    "    return pruned_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a CNN model-\n",
    "orig_model = pruned_nn(pruning_params_unpruned, pruning_params_unpruned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strip model of it's pruning parameters\n",
    "orig_model_stripped = sparsity.strip_pruning(orig_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save random weights-\n",
    "orig_model.save_weights(\"Conv_2_Devanagri_Random_Weights.h5\", overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_2 (Conv2D)            (None, 32, 32, 64)        640       \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 32, 32, 64)        36928     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 16, 16, 64)        0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 16384)             0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 256)               4194560   \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 256)               65792     \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 46)                11822     \n",
      "=================================================================\n",
      "Total params: 4,309,742\n",
      "Trainable params: 4,309,742\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Get CNN summary-\n",
    "orig_model_stripped.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 64400 samples, validate on 27600 samples\n",
      "Epoch 1/100\n",
      "64400/64400 [==============================] - 116s 2ms/sample - loss: 0.4587 - accuracy: 0.8674 - val_loss: 0.1825 - val_accuracy: 0.9445\n",
      "Epoch 2/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.1108 - accuracy: 0.9649 - val_loss: 0.1460 - val_accuracy: 0.9560\n",
      "Epoch 3/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.0592 - accuracy: 0.9806 - val_loss: 0.1244 - val_accuracy: 0.9639\n",
      "Epoch 4/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.0432 - accuracy: 0.9858 - val_loss: 0.1343 - val_accuracy: 0.9645\n",
      "Epoch 5/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.0349 - accuracy: 0.9888 - val_loss: 0.1029 - val_accuracy: 0.9727\n",
      "Epoch 6/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.0286 - accuracy: 0.9907 - val_loss: 0.1299 - val_accuracy: 0.9678\n",
      "Epoch 7/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.0278 - accuracy: 0.9912 - val_loss: 0.1388 - val_accuracy: 0.9668\n",
      "Epoch 8/100\n",
      "64400/64400 [==============================] - 114s 2ms/sample - loss: 0.0227 - accuracy: 0.9928 - val_loss: 0.1195 - val_accuracy: 0.9737\n"
     ]
    }
   ],
   "source": [
    "# Sanity check: Whether defined CNN works as intended-\n",
    "\n",
    "history_conv2 = orig_model.fit(\n",
    "    x = X_train, y = y_train,\n",
    "    epochs=num_epochs, batch_size=batch_size,\n",
    "    verbose=1, shuffle=True,\n",
    "    callbacks=callback,\n",
    "    validation_data=(X_test, y_test)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Trained Conv-2 CNN model metrics on validation set:\n",
      "val_loss = 0.11955 and val_accuracy = 97.36956%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Trained CNN performance on validation data set-\n",
    "model_val_loss, model_val_acc = orig_model.evaluate(X_test, y_test, verbose=0)\n",
    "\n",
    "print(\"\\nTrained Conv-2 CNN model metrics on validation set:\")\n",
    "print(\"val_loss = {0:.5f} and val_accuracy = {1:.5f}%\\n\".format(\n",
    "    model_val_loss, model_val_acc * 100\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pruned_nn(pruning_params_conv, pruning_params_fc, pruning_params_op):\n",
    "    \"\"\"\n",
    "    Function to define the architecture of a neural network model\n",
    "    following Conv-2 architecture for CIFAR-10 dataset and using\n",
    "    provided parameter which are used to prune the model.\n",
    "    \n",
    "    Conv-4 architecture-\n",
    "    64, 64, pool  -- convolutions\n",
    "    128, 128, pool -- convolutions\n",
    "    256, 256, 46  -- fully connected layers\n",
    "    \n",
    "    Input: 'pruning_params' Python 3 dictionary containing parameters which are used for pruning\n",
    "    Output: Returns designed and compiled neural network model\n",
    "    \"\"\"\n",
    "    \n",
    "    pruned_model = Sequential()\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 64, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same',\n",
    "            input_shape=(32, 32, 1)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "        \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 64, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 128, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 128, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    \n",
    "    pruned_model.add(Flatten())\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 256, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 256, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = num_classes, activation='softmax'\n",
    "        ),\n",
    "        **pruning_params_op)\n",
    "    )\n",
    "    \n",
    "\n",
    "    # Compile pruned CNN-\n",
    "    pruned_model.compile(\n",
    "        loss=tf.keras.losses.categorical_crossentropy,\n",
    "        # optimizer='adam',\n",
    "        optimizer=tf.keras.optimizers.Adam(lr = 0.0012),\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    \n",
    "    \n",
    "    return pruned_model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a CNN model-\n",
    "orig_model = pruned_nn(pruning_params_unpruned, pruning_params_unpruned, pruning_params_unpruned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strip model of it's pruning parameters\n",
    "orig_model_stripped = sparsity.strip_pruning(orig_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save random weights-\n",
    "orig_model.save_weights(\"Conv_4_Devanagri_Random_Weights.h5\", overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_4 (Conv2D)            (None, 32, 32, 64)        640       \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 32, 32, 64)        36928     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 16, 16, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 16, 16, 128)       73856     \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 16, 16, 128)       147584    \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 8, 8, 128)         0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 8192)              0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 256)               2097408   \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 256)               65792     \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 46)                11822     \n",
      "=================================================================\n",
      "Total params: 2,434,030\n",
      "Trainable params: 2,434,030\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Get CNN summary-\n",
    "orig_model_stripped.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 64400 samples, validate on 27600 samples\n",
      "Epoch 1/100\n",
      "64400/64400 [==============================] - 170s 3ms/sample - loss: 0.4297 - accuracy: 0.8769 - val_loss: 0.1237 - val_accuracy: 0.9615\n",
      "Epoch 2/100\n",
      "64400/64400 [==============================] - 167s 3ms/sample - loss: 0.0885 - accuracy: 0.9728 - val_loss: 0.0868 - val_accuracy: 0.9750\n",
      "Epoch 3/100\n",
      "64400/64400 [==============================] - 168s 3ms/sample - loss: 0.0569 - accuracy: 0.9818 - val_loss: 0.0734 - val_accuracy: 0.9790\n",
      "Epoch 4/100\n",
      "64400/64400 [==============================] - 168s 3ms/sample - loss: 0.0380 - accuracy: 0.9880 - val_loss: 0.0951 - val_accuracy: 0.9748\n",
      "Epoch 5/100\n",
      "64400/64400 [==============================] - 168s 3ms/sample - loss: 0.0351 - accuracy: 0.9892 - val_loss: 0.0672 - val_accuracy: 0.9814\n",
      "Epoch 6/100\n",
      "64400/64400 [==============================] - 168s 3ms/sample - loss: 0.0301 - accuracy: 0.9905 - val_loss: 0.0667 - val_accuracy: 0.9813\n",
      "Epoch 7/100\n",
      "64400/64400 [==============================] - 168s 3ms/sample - loss: 0.0226 - accuracy: 0.9934 - val_loss: 0.0911 - val_accuracy: 0.9805\n",
      "Epoch 8/100\n",
      "64400/64400 [==============================] - 168s 3ms/sample - loss: 0.0221 - accuracy: 0.9928 - val_loss: 0.0842 - val_accuracy: 0.9813\n"
     ]
    }
   ],
   "source": [
    "# Sanity check: Whether defined CNN works as intended-\n",
    "\n",
    "history_conv4 = orig_model.fit(\n",
    "    x = X_train, y = y_train,\n",
    "    epochs=num_epochs, batch_size=batch_size,\n",
    "    verbose=1, shuffle=True,\n",
    "    callbacks=callback,\n",
    "    validation_data=(X_test, y_test)\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Trained Conv-4 CNN model metrics on validation set:\n",
      "val_loss = 0.08423 and val_accuracy = 98.13043%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Trained CNN performance on validation data set-\n",
    "model_val_loss, model_val_acc = orig_model.evaluate(X_test, y_test, verbose=0)\n",
    "\n",
    "print(\"\\nTrained Conv-4 CNN model metrics on validation set:\")\n",
    "print(\"val_loss = {0:.5f} and val_accuracy = {1:.5f}%\\n\".format(\n",
    "    model_val_loss, model_val_acc * 100\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pruned_nn(pruning_params_conv, pruning_params_fc, pruning_params_op):\n",
    "    \"\"\"\n",
    "    Function to define the architecture of a neural network model\n",
    "    following Conv-6 architecture for CIFAR-10 dataset and using\n",
    "    provided parameter which are used to prune the model.\n",
    "    \n",
    "    Conv-6 architecture-\n",
    "    64, 64, pool  -- convolutional layers\n",
    "    128, 128, pool -- convolutional layers\n",
    "    256, 256, pool -- convolutional layers\n",
    "    256, 256, 46  -- fully connected layers\n",
    "    \n",
    "    Input: 'pruning_params' Python 3 dictionary containing parameters which are used for pruning\n",
    "    Output: Returns designed and compiled neural network model\n",
    "    \"\"\"\n",
    "    \n",
    "    pruned_model = Sequential()\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 64, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same',\n",
    "            input_shape=(32, 32, 1)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "        \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 64, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 128, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 128, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 256, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Conv2D(\n",
    "            filters = 256, kernel_size = (3, 3),\n",
    "            activation='relu', kernel_initializer = tf.initializers.GlorotUniform(),\n",
    "            strides = (1, 1), padding = 'same'\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "\n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        MaxPooling2D(\n",
    "            pool_size = (2, 2),\n",
    "            strides = (2, 2)\n",
    "        ),\n",
    "        **pruning_params_conv)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(Flatten())\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 256, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = 256, activation='relu',\n",
    "            kernel_initializer = tf.initializers.GlorotUniform()\n",
    "        ),\n",
    "        **pruning_params_fc)\n",
    "    )\n",
    "    \n",
    "    pruned_model.add(sparsity.prune_low_magnitude(\n",
    "        Dense(\n",
    "            units = num_classes, activation='softmax'\n",
    "        ),\n",
    "        **pruning_params_op)\n",
    "    )\n",
    "    \n",
    "\n",
    "    # Compile pruned CNN-\n",
    "    pruned_model.compile(\n",
    "        loss=tf.keras.losses.categorical_crossentropy,\n",
    "        # optimizer='adam',\n",
    "        optimizer=tf.keras.optimizers.Adam(lr = 0.0012),\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "    \n",
    "    \n",
    "    return pruned_model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize a CNN model-\n",
    "orig_model = pruned_nn(pruning_params_unpruned, pruning_params_unpruned, pruning_params_unpruned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strip model of it's pruning parameters-\n",
    "orig_model_stripped = sparsity.strip_pruning(orig_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save random weights-\n",
    "orig_model.save_weights(\"Conv_6_Devanagri_Random_Weights.h5\", overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_8 (Conv2D)            (None, 32, 32, 64)        640       \n",
      "_________________________________________________________________\n",
      "conv2d_9 (Conv2D)            (None, 32, 32, 64)        36928     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 16, 16, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_10 (Conv2D)           (None, 16, 16, 128)       73856     \n",
      "_________________________________________________________________\n",
      "conv2d_11 (Conv2D)           (None, 16, 16, 128)       147584    \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 8, 8, 128)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_12 (Conv2D)           (None, 8, 8, 256)         295168    \n",
      "_________________________________________________________________\n",
      "conv2d_13 (Conv2D)           (None, 8, 8, 256)         590080    \n",
      "_________________________________________________________________\n",
      "max_pooling2d_6 (MaxPooling2 (None, 4, 4, 256)         0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 4096)              0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 256)               1048832   \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 256)               65792     \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 46)                11822     \n",
      "=================================================================\n",
      "Total params: 2,270,702\n",
      "Trainable params: 2,270,702\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Get defined CNN summary-\n",
    "orig_model_stripped.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 64400 samples, validate on 27600 samples\n",
      "Epoch 1/100\n",
      "64400/64400 [==============================] - 225s 3ms/sample - loss: 0.7001 - accuracy: 0.8025 - val_loss: 0.1675 - val_accuracy: 0.9474\n",
      "Epoch 2/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.1062 - accuracy: 0.9667 - val_loss: 0.0796 - val_accuracy: 0.9776\n",
      "Epoch 3/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.0651 - accuracy: 0.9798 - val_loss: 0.1002 - val_accuracy: 0.9702\n",
      "Epoch 4/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.0545 - accuracy: 0.9830 - val_loss: 0.0731 - val_accuracy: 0.9788\n",
      "Epoch 5/100\n",
      "64400/64400 [==============================] - 223s 3ms/sample - loss: 0.0404 - accuracy: 0.9871 - val_loss: 0.0696 - val_accuracy: 0.9796\n",
      "Epoch 6/100\n",
      "64400/64400 [==============================] - 223s 3ms/sample - loss: 0.0347 - accuracy: 0.9890 - val_loss: 0.0677 - val_accuracy: 0.9801\n",
      "Epoch 7/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.0312 - accuracy: 0.9906 - val_loss: 0.0679 - val_accuracy: 0.9815\n",
      "Epoch 8/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.0279 - accuracy: 0.9916 - val_loss: 0.0628 - val_accuracy: 0.9843\n",
      "Epoch 9/100\n",
      "64400/64400 [==============================] - 223s 3ms/sample - loss: 0.0253 - accuracy: 0.9924 - val_loss: 0.0695 - val_accuracy: 0.9830\n",
      "Epoch 10/100\n",
      "64400/64400 [==============================] - 223s 3ms/sample - loss: 0.0246 - accuracy: 0.9926 - val_loss: 0.0562 - val_accuracy: 0.9857\n",
      "Epoch 11/100\n",
      "64400/64400 [==============================] - 223s 3ms/sample - loss: 0.0204 - accuracy: 0.9938 - val_loss: 0.0577 - val_accuracy: 0.9872\n",
      "Epoch 12/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.0205 - accuracy: 0.9938 - val_loss: 0.0776 - val_accuracy: 0.9829\n",
      "Epoch 13/100\n",
      "64400/64400 [==============================] - 222s 3ms/sample - loss: 0.0214 - accuracy: 0.9936 - val_loss: 0.0691 - val_accuracy: 0.9855\n"
     ]
    }
   ],
   "source": [
    "# Sanity check: Whether defined CNN works as intended-\n",
    "\n",
    "history_conv6 = orig_model.fit(\n",
    "    x = X_train, y = y_train,\n",
    "    epochs=num_epochs, batch_size=batch_size,\n",
    "    verbose=1, shuffle=True,\n",
    "    callbacks=callback,\n",
    "    validation_data=(X_test, y_test)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Trained Conv-6 CNN model metrics on validation set:\n",
      "val_loss = 0.06914 and val_accuracy = 98.55072%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Trained CNN performance on validation data set-\n",
    "model_val_loss, model_val_acc = orig_model.evaluate(X_test, y_test, verbose=0)\n",
    "\n",
    "print(\"\\nTrained Conv-6 CNN model metrics on validation set:\")\n",
    "print(\"val_loss = {0:.5f} and val_accuracy = {1:.5f}%\\n\".format(\n",
    "    model_val_loss, model_val_acc * 100\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion:\n",
    "\n",
    "It seems that _Conv-6_ Convolutional Neural Network (CNN) has the 'best' validation accuracy for the _Devanagri_ dataset. Therefore, _Conv-6_ CNN will be used for 'The Lottery Ticket Hypothesis'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}