NLP_C3W3_Modified_Triplet_Loss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Modified Triplet Loss : Ungraded Lecture Notebook\n",
    "In this notebook you'll see how to calculate the full triplet loss, step by step, including the mean negative and the closest negative. You'll also calculate the matrix of similarity scores.\n",
    "\n",
    "## Background\n",
    "The original triplet loss function looks like this:\n",
    "\n",
    "$\\mathcal{L_\\mathrm{Original}} = \\max{(\\mathrm{s}(A,N) -\\mathrm{s}(A,P) +\\alpha, 0)},$\n",
    "\n",
    "where the inputs are the Anchor $A$, Positive $P$ and Negative $N$. \n",
    "\n",
    "As you learned in the lectures, this loss can be improved by including the mean negative and the closest negative terms, to create a new full loss function. \n",
    "\n",
    "$\\mathcal{L_\\mathrm{1}} = \\max{(mean\\_neg -\\mathrm{s}(A,P)  +\\alpha, 0)}$\n",
    "\n",
    "$\\mathcal{L_\\mathrm{2}} = \\max{(closest\\_neg -\\mathrm{s}(A,P)  +\\alpha, 0)}$\n",
    "\n",
    "$\\mathcal{L_\\mathrm{Full}} = \\mathcal{L_\\mathrm{1}} + \\mathcal{L_\\mathrm{2}}$\n",
    "\n",
    "Let me show you what that means exactly, and how to calculate each step.\n",
    "\n",
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity Scores\n",
    "The first step is to calculate the matrix of similarity scores using cosine similarity so that you can look up $\\mathrm{s}(A,P)$, $\\mathrm{s}(A,N)$ as needed for the loss formulas.\n",
    "\n",
    "### Two Vectors\n",
    "First you will calculate the similarity score for 2 vectors using cosine similarity.\n",
    "\n",
    "$\\mathrm{s}(v_1,v_2) = \\mathrm{cosine \\ similarity}(v_1,v_2) = \\frac{v_1 \\cdot v_2}{||v_1||~||v_2||}$\n",
    "* Try changing the values in the second vector to see how it changes the cosine similarity.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Two vector example\n",
    "# Input data\n",
    "\n",
    "v1 = np.array([1, 2, 3], dtype=float)\n",
    "v2 = np.array([1, 2, 3.5], dtype=float)  # notice the 3rd element is offset by 0.5\n",
    "\n",
    "### START CODE HERE ###\n",
    "# Try modifying the vector v2 to see how it impacts the cosine similarity\n",
    "# v2 = v1                   # identical vector\n",
    "# v2 = v1 * -1              # opposite vector\n",
    "# v2 = np.array([0,-42,1], dtype=float)  # random example\n",
    "### END CODE HERE ###\n",
    "\n",
    "print(\"-- Inputs --\")\n",
    "print(\"v1 :\", v1)\n",
    "print(\"v2 :\", v2, \"\\n\")\n",
    "\n",
    "# Similarity score\n",
    "def cosine_similarity(v1, v2):\n",
    "    numerator = tf.math.reduce_sum(v1*v2) # takes the dot product between v1 and v2. Equivalent to np.dot(v1, v2)\n",
    "    denominator = tf.math.sqrt(tf.math.reduce_sum(v1*v1) * tf.math.reduce_sum(v2*v2))\n",
    "    return numerator / denominator\n",
    "\n",
    "print(\"-- Outputs --\")\n",
    "print(\"cosine similarity :\", cosine_similarity(v1, v2).numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe that here we are explicitly dividing by $\\sqrt{\\|{v_1}\\| \\|v_2\\|}$, to compute the cosine similarity. However, the output of the Siamese network as you have seen it so far includes a normalizing layer, so that $\\|v_1\\| = \\|v_2\\| = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two Batches of Vectors\n",
    "Now you will see how to calculate the similarity scores, using cosine similarity for 2 batches of vectors. These are rows of individual vectors, just like in the example above, but stacked vertically into a matrix. They would look like the image below for a batch size (row count) of 4 and embedding size (column count) of 5.\n",
    "\n",
    "The data is set up so that $v_{1\\_1}$ and $v_{2\\_1}$ represent duplicate inputs, but they are not duplicates with any other rows in the batch. This means $v_{1\\_1}$ and $v_{2\\_1}$ (green and green) have more similar vectors than say $v_{1\\_1}$ and $v_{2\\_2}$ (green and magenta).\n",
    "\n",
    "You will see two different methods for calculating the matrix of similarities from 2 batches of vectors.\n",
    "\n",
    "<img src = 'images/v1v2_stacked.png' width=\"width\" height=\"height\" style=\"height:250px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First you will create the similarity matrix for batches $v_1$ and $v_2$, filling each element of the matrix at a time. This involves two nested `for` loops, which isn't very efficient. However it is very pedagogic, since you get to see how each element of the similarity matrix is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "lines_to_next_cell": 2,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Two batches of vectors example\n",
    "# Input data\n",
    "\n",
    "v1_1 = np.array([1.0, 2.0, 3.0])\n",
    "v1_2 = np.array([9.0, 8.0, 7.0])\n",
    "v1_3 = np.array([-1.0, -4.0, -2.0])\n",
    "v1_4 = np.array([1.0, -7.0, 2.0])\n",
    "v1 = np.vstack([v1_1, v1_2, v1_3, v1_4])\n",
    "\n",
    "v2_1 = v1_1 + np.random.normal(0, 2, 3)  # add some noise to create approximate duplicate\n",
    "v2_2 = v1_2 + np.random.normal(0, 2, 3)\n",
    "v2_3 = v1_3 + np.random.normal(0, 2, 3)\n",
    "v2_4 = v1_4 + np.random.normal(0, 2, 3)\n",
    "v2 = np.vstack([v2_1, v2_2, v2_3, v2_4])\n",
    "\n",
    "print(\"-- Inputs --\")\n",
    "print(f\"v1 :\\n{v1}\\n\")\n",
    "print(f\"v2 :\\n{v2}\\n\")\n",
    "\n",
    "# Batch sizes must match\n",
    "b = len(v1)\n",
    "print(f\"Batch sizes match : {b == len(v2)}\\n\")\n",
    "\n",
    "# Similarity scores\n",
    "\n",
    "# Option 1 : nested loops and the cosine similarity function\n",
    "sim_1 = np.zeros([b, b])  # empty array to take similarity scores\n",
    "# Loop\n",
    "for row in range(0, sim_1.shape[0]):\n",
    "    for col in range(0, sim_1.shape[1]):\n",
    "        sim_1[row, col] = cosine_similarity(v2[row], v1[col]).numpy()\n",
    "\n",
    "print(\"-- Outputs --\")\n",
    "print(\"Option 1 : loop\")\n",
    "print(sim_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can repeat the procedure applying vectorization, so the computations are more efficient. For this small example you will not notice a difference, but for training a big model this is crucial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 2 : vector normalization and dot product\n",
    "def norm(x):\n",
    "    return tf.math.l2_normalize(x, axis=1) # use tensorflow built in normalization\n",
    "\n",
    "sim_2 = tf.linalg.matmul(norm(v2), norm(v1), transpose_b=True)\n",
    "\n",
    "print(\"-- Outputs --\")\n",
    "print(\"Option 2 : vector normalization and dot product\")\n",
    "print(sim_2, \"\\n\")\n",
    "\n",
    "# Check\n",
    "print(f\"Outputs are the same : {np.allclose(sim_1, sim_2)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hard Negative Mining\n",
    "\n",
    "You will now calculate the mean negative $mean\\_neg$ and the closest negative $close\\_neg$ used in calculating $\\mathcal{L_\\mathrm{1}}$ and $\\mathcal{L_\\mathrm{2}}$.\n",
    "\n",
    "\n",
    "$\\mathcal{L_\\mathrm{1}} = \\max{(mean\\_neg -\\mathrm{s}(A,P)  +\\alpha, 0)}$\n",
    "\n",
    "$\\mathcal{L_\\mathrm{2}} = \\max{(closest\\_neg -\\mathrm{s}(A,P)  +\\alpha, 0)}$\n",
    "\n",
    "You'll do this using the matrix of similarity scores you already know how to make, like the example below for a batch size of 4. The diagonal of the matrix contains all the $\\mathrm{s}(A,P)$ values, similarities from duplicate question pairs (aka Positives). This is an important attribute for the calculations to follow.\n",
    "\n",
    "<img src = 'images/ss_matrix.png' width=\"width\" height=\"height\" style=\"height:250px;\"/>\n",
    "\n",
    "\n",
    "### Mean Negative\n",
    "$mean\\_neg$ is the average of the off diagonals, the $\\mathrm{s}(A,N)$ values, for each row.\n",
    "\n",
    "### Closest Negative\n",
    "$closest\\_neg$ is the largest off diagonal value, $\\mathrm{s}(A,N)$, that is smaller than the diagonal $\\mathrm{s}(A,P)$ for each row.\n",
    "* Try using a different matrix of similarity scores. \n",
    "\n",
    "First, try the implementation in `NumPy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Hardcoded matrix of similarity scores\n",
    "sim_hardcoded = np.array(\n",
    "    [\n",
    "        [0.9, -0.8, 0.3, -0.5],\n",
    "        [-0.4, 0.5, 0.1, -0.1],\n",
    "        [0.3, 0.1, -0.4, -0.8],\n",
    "        [-0.5, -0.2, -0.7, 0.5],\n",
    "    ]\n",
    ")\n",
    "\n",
    "sim = sim_hardcoded\n",
    "\n",
    "### START CODE HERE ###\n",
    "# Try using different values for the matrix of similarity scores\n",
    "# sim = 2 * np.random.random_sample((b,b)) -1   # random similarity scores between -1 and 1\n",
    "# sim = sim_2                                   # the matrix calculated previously using vector normalization and dot product\n",
    "### END CODE HERE ###\n",
    "\n",
    "# Batch size\n",
    "b = sim.shape[0]\n",
    "\n",
    "print(\"-- Inputs --\")\n",
    "print(f\"sim:\")\n",
    "print(sim)\n",
    "print(f\"shape: {sim.shape}\\n\")\n",
    "\n",
    "# Positives\n",
    "# All the s(A,P) values : similarities from duplicate question pairs (aka Positives)\n",
    "# These are along the diagonal\n",
    "sim_ap = np.diag(sim)\n",
    "print(\"sim_ap:\")\n",
    "print(np.diag(sim_ap))\n",
    "\n",
    "\n",
    "# Negatives\n",
    "# all the s(A,N) values : similarities the non duplicate question pairs (aka Negatives)\n",
    "# These are in the off diagonals\n",
    "sim_an = sim - np.diag(sim_ap)\n",
    "print(\"\\nsim_an:\")\n",
    "print(sim_an)\n",
    "\n",
    "print(\"\\n-- Outputs --\")\n",
    "# Mean negative\n",
    "# Average of the s(A,N) values for each row\n",
    "mean_neg = np.sum(sim_an, axis=1, keepdims=True) / (b - 1)\n",
    "print(\"\\nmean_neg:\")\n",
    "print(mean_neg)\n",
    "\n",
    "# Closest negative\n",
    "# Max s(A,N) that is <= s(A,P) for each row\n",
    "mask_1 = np.identity(b) == 1            # mask to exclude the diagonal\n",
    "mask_2 = sim_an > sim_ap.reshape(b, 1)  # mask to exclude sim_an > sim_ap\n",
    "mask = mask_1 | mask_2\n",
    "sim_an_masked = np.copy(sim_an)         # create a copy to preserve sim_an\n",
    "sim_an_masked[mask] = -2\n",
    "\n",
    "closest_neg = np.max(sim_an_masked, axis=1, keepdims=True)\n",
    "print(\"\\nclosest_neg :\")\n",
    "print(closest_neg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now have a look at the implementation in `TensorFlow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hardcoded matrix of similarity scores\n",
    "sim_hardcoded = np.array(\n",
    "    [\n",
    "        [0.9, -0.8, 0.3, -0.5],\n",
    "        [-0.4, 0.5, 0.1, -0.1],\n",
    "        [0.3, 0.1, -0.4, -0.8],\n",
    "        [-0.5, -0.2, -0.7, 0.5],\n",
    "    ]\n",
    ")\n",
    "\n",
    "sim = sim_hardcoded\n",
    "\n",
    "### START CODE HERE ###\n",
    "# Try using different values for the matrix of similarity scores\n",
    "# sim = 2 * np.random.random_sample((b,b)) -1   # random similarity scores between -1 and 1\n",
    "# sim = sim_2                                   # the matrix calculated previously using vector normalization and dot product\n",
    "### END CODE HERE ###\n",
    "\n",
    "# Batch size\n",
    "b = sim.shape[0]\n",
    "\n",
    "print(\"-- Inputs --\")\n",
    "print(\"sim :\")\n",
    "print(sim)\n",
    "print(\"shape :\", sim.shape, \"\\n\")\n",
    "\n",
    "# Positives\n",
    "# All the s(A,P) values : similarities from duplicate question pairs (aka Positives)\n",
    "# These are along the diagonal\n",
    "sim_ap = tf.linalg.diag_part(sim) # this is just a 1D array of diagonal elements\n",
    "print(\"sim_ap :\")\n",
    "# tf.linalg.diag makes a diagonal matrix given an array\n",
    "print(tf.linalg.diag(sim_ap), \"\\n\")\n",
    "\n",
    "# Negatives\n",
    "# all the s(A,N) values : similarities the non duplicate question pairs (aka Negatives)\n",
    "# These are in the off diagonals\n",
    "sim_an = sim - tf.linalg.diag(sim_ap)\n",
    "print(\"sim_an :\")\n",
    "print(sim_an, \"\\n\")\n",
    "\n",
    "print(\"-- Outputs --\")\n",
    "# Mean negative\n",
    "# Average of the s(A,N) values for each row\n",
    "mean_neg = tf.math.reduce_sum(sim_an, axis=1) / (b - 1)\n",
    "print(\"mean_neg :\")\n",
    "print(mean_neg, \"\\n\")\n",
    "\n",
    "# Closest negative\n",
    "# Max s(A,N) that is <= s(A,P) for each row\n",
    "mask_1 = tf.eye(b) == 1            # mask to exclude the diagonal\n",
    "mask_2 = sim_an > tf.expand_dims(sim_ap, 1)  # mask to exclude sim_an > sim_ap\n",
    "mask = tf.cast(mask_1 | mask_2, tf.float64)\n",
    "sim_an_masked = sim_an - 2.0*mask\n",
    "\n",
    "closest_neg = tf.math.reduce_max(sim_an_masked, axis=1)\n",
    "print(\"closest_neg :\")\n",
    "print(closest_neg, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Loss Functions\n",
    "\n",
    "The last step is to calculate the loss functions.\n",
    "\n",
    "$\\mathcal{L_\\mathrm{1}} = \\max{(mean\\_neg -\\mathrm{s}(A,P)  +\\alpha, 0)}$\n",
    "\n",
    "$\\mathcal{L_\\mathrm{2}} = \\max{(closest\\_neg -\\mathrm{s}(A,P)  +\\alpha, 0)}$\n",
    "\n",
    "$\\mathcal{L_\\mathrm{Full}} = \\mathcal{L_\\mathrm{1}} + \\mathcal{L_\\mathrm{2}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Alpha margin\n",
    "alpha = 0.25\n",
    "\n",
    "# Modified triplet loss\n",
    "# Loss 1\n",
    "l_1 = tf.maximum(mean_neg - sim_ap + alpha, 0)\n",
    "print(f\"Loss 1: {l_1}\\n\")\n",
    "# Loss 2\n",
    "l_2 = tf.maximum(closest_neg - sim_ap + alpha, 0)\n",
    "print(f\"Loss 2: {l_2}\\n\")\n",
    "# Loss full<\n",
    "l_full = l_1 + l_2\n",
    "# Cost\n",
    "cost = tf.math.reduce_sum(l_full)\n",
    "\n",
    "print(\"-- Outputs --\")\n",
    "print(\"Loss full :\")\n",
    "print(l_full, \"\\n\")\n",
    "print(\"Cost :\", \"{:.3f}\".format(cost))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "There were a lot of steps in there, so well done. You can now calculate a modified triplet loss, incorporating the mean negative and the closest negative. You also learned how to create a matrix of similarity scores based on cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}