NLP_C3W3_Accuracy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate a Siamese model: Ungraded Lecture Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import tensorflow.math as math\n",
    "import tensorflow.linalg as linalg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting the necessary elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lecture notebook you will learn how to evaluate a Siamese model using the accuracy metric. Because there are many steps before evaluating a Siamese network (as you will see in this week's assignment) the necessary elements and variables are replicated here using real data from the assignment:\n",
    "\n",
    "   - `q1`: vector with dimension `(batch_size X max_length)` containing first questions to compare in the test set.\n",
    "   - `q2`: vector with dimension `(batch_size X max_length)` containing second questions to compare in the test set.\n",
    "   \n",
    "   **Notice that for each pair of vectors within a batch $([q1_1, q1_2, q1_3, ...]$, $[q2_1, q2_2,q2_3, ...])$  $q1_i$ is associated to $q2_k$.**\n",
    "        \n",
    "        \n",
    "   - `y_test`: 1 if  $q1_i$ and $q2_k$ are duplicates, 0 otherwise.\n",
    "   \n",
    "   - `v1`: output vector from the model's prediction associated with the first questions.\n",
    "   - `v2`: output vector from the model's prediction associated with the second questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can inspect each one of these variables by running the following cells:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "q1 = np.load('./data/q1.npy')\n",
    "print(f'q1 has shape: {q1.shape} \\n\\nAnd it looks like this: \\n\\n {q1}\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice those 1s on the right-hand side?  \n",
    "\n",
    "Hope you remember that the value of `1` was used for padding. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "q2 = np.load('./data/q2.npy')\n",
    "print(f'q2 has shape: {q2.shape} \\n\\nAnd looks like this: \\n\\n {q2}\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = np.load('./data/y_test.npy')\n",
    "print(f'y_test has shape: {y_test.shape} \\n\\nAnd looks like this: \\n\\n {y_test}\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = np.load('./data/v1.npy')\n",
    "print(f'v1 has shape: {v1.shape} \\n\\nAnd looks like this: \\n\\n {v1}\\n\\n')\n",
    "v2 = np.load('./data/v2.npy')\n",
    "print(f'v2 has shape: {v2.shape} \\n\\nAnd looks like this: \\n\\n {v2}\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the accuracy\n",
    "\n",
    "You will calculate the accuracy by iterating over the test set and checking if the model predicts right or wrong.\n",
    "\n",
    "The first step is to set the accuracy to zero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will also need the `batch size` and the `threshold` that determines if two questions are the same or not. \n",
    "\n",
    "**Note :A higher threshold means that only very similar questions will be considered as the same question.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 512 # Note: The max it can be is y_test.shape[0] i.e all the samples in test data\n",
    "threshold = 0.7 # You can play around with threshold and then see the change in accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the assignment you will iterate over multiple batches of data but since this is a simplified version only one batch is provided. \n",
    "\n",
    "**Note: Be careful with the indices when slicing the test data in the assignment!**\n",
    "\n",
    "The process is pretty straightforward:\n",
    "   - Iterate over each one of the elements in the batch\n",
    "   - Compute the cosine similarity between the predictions\n",
    "       - For computing the cosine similarity, the two output vectors should have been normalized using L2 normalization meaning their magnitude will be 1. This has been taken care of by the Siamese network you will build in the assignment. Hence the cosine similarity here is just dot product between two vectors. You can check by implementing the usual cosine similarity formula and check if this holds or not.\n",
    "   - Determine if this value is greater than the threshold (If it is, consider the two questions as the same and return 1 else 0)\n",
    "   - Compare against the actual target and if the prediction matches, add 1 to the accuracy (increment the correct prediction counter)\n",
    "   - Divide the accuracy by the number of processed elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for j in range(batch_size):        # Iterate over each element in the batch\n",
    "    d = math.reduce_sum(v1[j]*v2[j])# Compute the cosine similarity between the predictions as l2 normalized, ||v1[j]||==||v2[j]||==1 so only dot product is needed\n",
    "    res = d > threshold            # Determine if this value is greater than the threshold (if it is consider the two questions as the same)\n",
    "    accuracy += tf.cast(y_test[j] == res, tf.int32) # Compare against the actual target and if the prediction matches, add 1 to the accuracy\n",
    "\n",
    "accuracy = accuracy / batch_size   # Divide the accuracy by the number of processed elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'The accuracy of the model is: {accuracy}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This implementation with the `for` loop is great for understanding exactly what is going on when you are computing the accuracy, however you can implement this in a more efficient way using vectorization. You will do this in the assignment.\n",
    "\n",
    "Feel free to add a new cell and try it out with this small dataset, and check that you are getting the same results. You can do this with exactly the same functions you used in the cell above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing this lecture notebook!** \n",
    "\n",
    "Now you should have a clearer understanding of how to evaluate your Siamese language models using the accuracy metric. \n",
    "\n",
    "**Keep it up!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}