U_net_tutorial_with_transfer_learning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/williamlidberg/Unet-tutorial/blob/main/U_net_tutorial_with_transfer_learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NppLzYcwYzMx"
      },
      "source": [
        "# Intro to google colab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7HiFfWUAY1Xa"
      },
      "source": [
        "# Cells\n",
        "A notebook is a list of cells. Cells contain either explanatory text or executable code and its output. Click a cell to select it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBpis6_FY28V"
      },
      "source": [
        "## Code cells\n",
        "Below is a **code cell**. Once the toolbar button indicates CONNECTED, click in the cell to select it and execute the contents in the following ways:\n",
        "\n",
        "* Click the **Play icon** in the left gutter of the cell;\n",
        "* Type **Cmd/Ctrl+Enter** to run the cell in place;\n",
        "* Type **Shift+Enter** to run the cell and move focus to the next cell (adding one if none exists); or\n",
        "* Type **Alt+Enter** to run the cell and insert a new code cell immediately below it.\n",
        "\n",
        "There are additional options for running some or all cells in the **Runtime** menu.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DOKeOQKcYzhd"
      },
      "outputs": [],
      "source": [
        "a = 10\n",
        "a"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UGKhzb7qY7J4"
      },
      "source": [
        "# Working with python\n",
        "Colaboratory is built on top of [Jupyter Notebook](https://jupyter.org/). Below are some examples of convenience functions provided."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cBjnXfd7Y83S"
      },
      "outputs": [],
      "source": [
        "import time\n",
        "print(\"Sleeping\")\n",
        "time.sleep(30) # sleep for a while; interrupt me!\n",
        "print(\"Done Sleeping\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ktwsjs5HZAal"
      },
      "source": [
        "## Adding and moving cells\n",
        "You can add new cells by using the **+ CODE** and **+ TEXT** buttons that show when you hover between cells. These buttons are also in the toolbar above the notebook where they can be used to add a cell below the currently selected cell.\n",
        "\n",
        "You can move a cell by selecting it and clicking **Cell Up** or **Cell Down** in the top toolbar.\n",
        "\n",
        "Consecutive cells can be selected by \"lasso selection\" by dragging from outside one cell and through the group.  Non-adjacent cells can be selected concurrently by clicking one and then holding down Ctrl while clicking another.  Similarly, using Shift instead of Ctrl will select all intermediate cells."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kml97TcLZF5j"
      },
      "source": [
        "# Deep learning for cultural remains\n",
        "In this tutorial you will learn:\n",
        "\n",
        "\n",
        "1.   Google colab\n",
        "2.   Import python libraries\n",
        "3.   Run python code\n",
        "6.   Build a Unet model\n",
        "7.   Train and test a Unet model\n",
        "8.   Transfer learning\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFb2Ee2OhgsM"
      },
      "source": [
        "## Import data\n",
        "These two lines downloads a dataset from williams google drive. The first line is long and complicated due to googles shenanigans. The second line unpacks the data into google colabs file tree."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KGd8uz_ThiP_"
      },
      "outputs": [],
      "source": [
        "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1EQKhlQFjg0mP7OfoT0NRURQZAgRAqBvj' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1EQKhlQFjg0mP7OfoT0NRURQZAgRAqBvj\" -O /content/earth.zip && rm -rf /tmp/cookies.txt\n",
        "!unzip /content/earth.zip -d /content/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v1Rbm_5yZIyP"
      },
      "source": [
        "## Import libraries\n",
        "Python libraries are like R packages. Before they can be used they need to be imported into the python environment. This is how you install a python package."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install visualkeras"
      ],
      "metadata": {
        "id": "iip2Ex4Dc9WL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JrOZl2ppZGQl"
      },
      "outputs": [],
      "source": [
        "import os # For data handling\n",
        "import glob # For data handling\n",
        "import numpy as np # For arrays\n",
        "import keras\n",
        "import visualkeras\n",
        "import tensorflow.keras.backend as K # For deep learning\n",
        "import tensorflow as tf # For deep learning\n",
        "import matplotlib.pyplot as plt # For plotting data\n",
        "import random # for seeds\n",
        "random.seed(10) # To increase reproducability"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "COuJVWg1dSol"
      },
      "source": [
        "## Inspect data\n",
        "The first stepp in any data analysis is always to inspect the data. The data in this tutorial is stored as a tensorflow dataset. If you want to get into the details on how to create a training dataset you can check out the github linked to the research paper: https://github.com/williamlidberg/Detection-of-hunting-pits-using-airborne-laser-scanning-and-deep-learning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uSkqPX4ZFkhC"
      },
      "outputs": [],
      "source": [
        "train_images = tf.data.Dataset.load('/content/earth/train/') # These will be used to train the model\n",
        "test_images = tf.data.Dataset.load('/content/earth/test/') # These will be used to test the model to get a estimate of the preformance.\n",
        "train_examples = list(train_images.as_numpy_iterator())\n",
        "print(len(train_images), 'images will be used to train the model.')\n",
        "print(len(test_images), 'images will be used to test the model.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zfAW-XLjGZ0Q"
      },
      "source": [
        "Plot some examples from the training data to make sure it looks resonable. Hunting pits a tiny feaures and on the limit of what can be seen in this digital elevation model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YNlvEetyk4bw"
      },
      "outputs": [],
      "source": [
        "example1 = np.squeeze(train_examples[0]) # image number 0\n",
        "image1 = example1[0,:,:,]\n",
        "label1 = example1[1,:,:,]\n",
        "\n",
        "example2 = np.squeeze(train_examples[42]) # image number 42\n",
        "image2 = example2[0,:,:,]\n",
        "label2 = example2[1,:,:,]\n",
        "\n",
        "example3 = np.squeeze(train_examples[1000])  # image number 1000\n",
        "image3 = example3[0,:,:,]\n",
        "label3 = example3[1,:,:,]\n",
        "\n",
        "# select and plot images\n",
        "plt.rcParams['figure.figsize'] = [12, 12]\n",
        "f, axarr = plt.subplots(3,2)\n",
        "axarr[0,0].set_title('Hillshade')\n",
        "axarr[0,0].imshow(image1, cmap='Greys_r')\n",
        "axarr[0,1].set_title('Labeled hunting pit')\n",
        "axarr[0,1].imshow(label1,cmap='Greys')\n",
        "\n",
        "axarr[1,0].set_title('Hillshade')\n",
        "axarr[1,0].imshow(image2, cmap='Greys_r')\n",
        "axarr[1,1].set_title('Labeled hunting pit')\n",
        "axarr[1,1].imshow(label2,cmap='Greys')\n",
        "\n",
        "axarr[2,0].set_title('Hillshade')\n",
        "axarr[2,0].imshow(image3, cmap='Greys_r')\n",
        "axarr[2,1].set_title('Labeled hunting pit')\n",
        "axarr[2,1].imshow(label3,cmap='Greys')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "909q_BDzgbFK"
      },
      "source": [
        "## Build input pipelines\n",
        "[Batch size](https://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57) is a term used in machine learning and refers to the number of training examples utilized in one iteration. Larger batch sizes is genreally better but GPU memory usually very limited.\n",
        "\n",
        "Buffer size is how many images that should be read into system memory (RAM) at the time. For a large number of images it is not possible to read the entire dataset into the system memory but having a small buffer ready speeds up the training process. Otherwise the images needs to be read from disk which is much slower."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Sylbq54hQVf"
      },
      "source": [
        "**Training pipeline**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pINwJIaXf5Aj"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 16\n",
        "BUFFER_SIZE = 32\n",
        "train_batches = (train_images\n",
        "                    .cache() # cache data\n",
        "                    .shuffle(BUFFER_SIZE) # fill buffer, sample from it and replace with new items (buffer size > training set for perfect shuffling)\n",
        "                    .batch(BATCH_SIZE)  # set batch size\n",
        "                    .repeat()  # repeat dataset idefinetely\n",
        "                    .prefetch(buffer_size=32)) # prefetch 100 images to optimize runtime"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kI7WUPJEhTGY"
      },
      "source": [
        "**Testing pipeline**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Eu_g0qswhVW0"
      },
      "outputs": [],
      "source": [
        "# setup input pipeline\n",
        "BATCH_SIZE = 16\n",
        "BUFFER_SIZE = 32\n",
        "test_batches = (test_images\n",
        "                    .cache() # cache data\n",
        "                    .shuffle(BUFFER_SIZE) # fill buffer, sample from it and replace with new items (buffer size > training set for perfect shuffling)\n",
        "                    .batch(BATCH_SIZE)  # set batch size\n",
        "                    .repeat(1)  # repeat dataset idefinetely\n",
        "                    .prefetch(buffer_size=32)) # prefetch 100 images to optimize runtime"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zM_24YgsgdQd"
      },
      "source": [
        "## Define metrics to use for evaluation\n",
        "Accuracy is how many of the pixels are classified correctly by the model. This is not very usefull if most of the landscape consists of one class. In this case most of the forest are not hunting pits.\n",
        "\n",
        "f1 score is a better option since it tries to take the inbalance of the data into account. f1 = 0 is very low and f1 = 1 is a perfect fit. Both recall and precision are required to calculate F1 score so these have to be defined as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-G4M_u8kgelb"
      },
      "outputs": [],
      "source": [
        "def recall_m(y_true, y_pred):\n",
        "    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
        "    recall = true_positives / (possible_positives + K.epsilon())\n",
        "    return recall\n",
        "\n",
        "def precision_m(y_true, y_pred):\n",
        "    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
        "    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
        "    precision = true_positives / (predicted_positives + K.epsilon())\n",
        "    return precision\n",
        "\n",
        "def f1_m(y_true, y_pred):\n",
        "    precision = precision_m(y_true, y_pred)\n",
        "    recall = recall_m(y_true, y_pred)\n",
        "    return 2*((precision*recall)/(precision+recall+K.epsilon()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G7SNFqh3gtEx"
      },
      "source": [
        "## Build U-net model\n",
        "[U-net](https://arxiv.org/abs/1505.04597) is a convolutional neural network that was developed for biomedical image segmentation. The network consists of a contracting path and an expansive path, which gives it the [u-shaped architecture](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/). The contracting path is a typical convolutional network that consists of repeated application of convolutions, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. The expansive pathway combines the feature and spatial information through a sequence of up-convolutions and concatenations with high-resolution features from the contracting path."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aW3gKjhNgtzY"
      },
      "outputs": [],
      "source": [
        "#Build the model\n",
        "IMG_WIDTH = 256 # image width in number of pixels\n",
        "IMG_HEIGHT = 256 # image height in number of pixels\n",
        "IMG_CHANNELS = 1 # a grey scale image only has one band. Normal images (RGB) have three channels.\n",
        "NUM_CLASSES = 1 # The images are labaled into 0 and 1 where 0 = no hunting pit and 1 = hunting pit\n",
        "\n",
        "inputs = tf.keras.layers.Input((IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS))\n",
        "\n",
        "#Contraction path\n",
        "c1 = tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(inputs) # 32 is the number of feature maps and 3, 3 is the size of the convulutional filter.\n",
        "c1 = tf.keras.layers.Dropout(0.1)(c1) # dropouts combats overfitting\n",
        "c1 = tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c1)\n",
        "p1 = tf.keras.layers.MaxPooling2D((2, 2))(c1)\n",
        "\n",
        "c2 = tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(p1)\n",
        "c2 = tf.keras.layers.Dropout(0.1)(c2)\n",
        "c2 = tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c2)\n",
        "p2 = tf.keras.layers.MaxPooling2D((2, 2))(c2)\n",
        "\n",
        "c3 = tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(p2)\n",
        "c3 = tf.keras.layers.Dropout(0.2)(c3)\n",
        "c3 = tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c3)\n",
        "p3 = tf.keras.layers.MaxPooling2D((2, 2))(c3)\n",
        "\n",
        "c4 = tf.keras.layers.Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(p3)\n",
        "c4 = tf.keras.layers.Dropout(0.2)(c4)\n",
        "c4 = tf.keras.layers.Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c4)\n",
        "p4 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(c4)\n",
        "\n",
        "c5 = tf.keras.layers.Conv2D(512, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(p4)\n",
        "c5 = tf.keras.layers.Dropout(0.3)(c5)\n",
        "c5 = tf.keras.layers.Conv2D(512, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c5)\n",
        "\n",
        "#Expansive path\n",
        "u6 = tf.keras.layers.Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(c5)\n",
        "u6 = tf.keras.layers.concatenate([u6, c4])\n",
        "c6 = tf.keras.layers.Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(u6)\n",
        "c6 = tf.keras.layers.Dropout(0.2)(c6)\n",
        "c6 = tf.keras.layers.Conv2D(256, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c6)\n",
        "\n",
        "u7 = tf.keras.layers.Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(c6)\n",
        "u7 = tf.keras.layers.concatenate([u7, c3])\n",
        "c7 = tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(u7)\n",
        "c7 = tf.keras.layers.Dropout(0.2)(c7)\n",
        "c7 = tf.keras.layers.Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c7)\n",
        "\n",
        "u8 = tf.keras.layers.Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(c7)\n",
        "u8 = tf.keras.layers.concatenate([u8, c2])\n",
        "c8 = tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(u8)\n",
        "c8 = tf.keras.layers.Dropout(0.1)(c8)\n",
        "c8 = tf.keras.layers.Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c8)\n",
        "\n",
        "u9 = tf.keras.layers.Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(c8)\n",
        "u9 = tf.keras.layers.concatenate([u9, c1], axis=3)\n",
        "c9 = tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(u9)\n",
        "c9 = tf.keras.layers.Dropout(0.1)(c9)\n",
        "c9 = tf.keras.layers.Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_normal', padding='same')(c9)\n",
        "\n",
        "outputs = tf.keras.layers.Conv2D(NUM_CLASSES, (1, 1))(c9)\n",
        "\n",
        "model = tf.keras.Model(inputs=[inputs], outputs=[outputs])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iuV21UdehnFb"
      },
      "source": [
        "We can also print out a text summary of the model. Note that the first and last layers are both 256x256 just like our resized data. The 1 in the first layer represents the number of channels. A RGB image would have 3 channels but since this is a greyscale it only has 1."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "id": "8HbsQCULLCc3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "That might be a bit dense to understand as a beginer. Another alternative is to make a figure of the model instead."
      ],
      "metadata": {
        "id": "w_a7gltQdVjf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "visualkeras.layered_view(model, legend=True)"
      ],
      "metadata": {
        "id": "WK74UpGzdb_f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pXfbgChXh_rd"
      },
      "source": [
        "## Train the U-net\n",
        "Now we are ready to train the UNet. Epoch is one pass of the model across the training data. Training the model for more epochs takes longer time but gives more accurate results.\n",
        "\n",
        "The “steps per epoch” parameter is a value that defines the total number of steps taken before one epoch is finished and the next epoch starts during training. It is typically set to the total number of samples in your dataset divided by the batch size. 10 epochs takes around 5min to train on Colabs standard GPU."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JMyqUlh3rkqb"
      },
      "outputs": [],
      "source": [
        "steps = len(train_images)/BATCH_SIZE # len means number of images in the training data. Batch size was defined above.\n",
        "model.compile(tf.keras.optimizers.Adam(learning_rate=0.001), loss=tf.keras.losses.BinaryFocalCrossentropy(gamma=2.0, from_logits=True), metrics=['acc', f1_m, recall_m])\n",
        "result = model.fit(train_batches, epochs=10, steps_per_epoch=steps)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYEChOQTi6-Z"
      },
      "source": [
        "## Plot the training history"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K712GpdXi_Hf"
      },
      "outputs": [],
      "source": [
        "# summarize history for accuracy\n",
        "plt.rcParams['figure.figsize'] = [8, 8]\n",
        "plt.plot(result.history['f1_m'])\n",
        "plt.title('Model accuracy using f1 score')\n",
        "plt.ylabel('f1 score')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')\n",
        "plt.xlim(0,10)\n",
        "plt.ylim(0,1)\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbrtqQbiiMdw"
      },
      "source": [
        "## Evaluate the model by applying it to the test images"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tdBAFswGiL4J"
      },
      "outputs": [],
      "source": [
        "results = model.evaluate(test_batches)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "li57W8-mqPRJ"
      },
      "source": [
        "The model seems to strugle to learn how to detect hunting pits. Probably due to a combination of a low number of training images and the fact that hunting pits are very small."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8yMfYOAhFGuO"
      },
      "outputs": [],
      "source": [
        "plt.rcParams['figure.figsize'] = [16, 16]\n",
        "f, axarr = plt.subplots(1,3)\n",
        "\n",
        "test_examples = list(test_images.as_numpy_iterator())\n",
        "test_example1 = np.squeeze(test_examples[1])\n",
        "original = test_example1[0,:,:,]\n",
        "label = test_example1[1,:,:,]\n",
        "\n",
        "testimage = np.reshape(original,[1,256,256,1])\n",
        "prediction = model.predict(testimage)\n",
        "pred = np.squeeze(prediction)\n",
        "\n",
        "# Image\n",
        "axarr[0].set_title('Original image')\n",
        "axarr[0].imshow(original, cmap='Greys_r')\n",
        "\n",
        "# Label\n",
        "axarr[1].set_title('Labeled pits')\n",
        "axarr[1].imshow(label, cmap='Greys_r')\n",
        "\n",
        "# Prediction\n",
        "axarr[2].set_title('Predicted pits')\n",
        "axarr[2].imshow(pred)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can set pred>0.5 to get a binary prediction just like the label"
      ],
      "metadata": {
        "id": "eph9jGZKPIFL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.rcParams['figure.figsize'] = [16, 16]\n",
        "f, axarr = plt.subplots(1,3)\n",
        "\n",
        "test_examples = list(test_images.as_numpy_iterator())\n",
        "test_example1 = np.squeeze(test_examples[1])\n",
        "original = test_example1[0,:,:,]\n",
        "label = test_example1[1,:,:,]\n",
        "\n",
        "testimage = np.reshape(original,[1,256,256,1])\n",
        "prediction = model.predict(testimage)\n",
        "pred = np.squeeze(prediction)\n",
        "\n",
        "# Image\n",
        "axarr[0].set_title('Original image')\n",
        "axarr[0].imshow(original, cmap='Greys_r')\n",
        "\n",
        "# Label\n",
        "axarr[1].set_title('Labeled pits')\n",
        "axarr[1].imshow(label, cmap='Greys_r')\n",
        "\n",
        "# Prediction\n",
        "axarr[2].set_title('Predicted pits')\n",
        "axarr[2].imshow(pred>0.5, cmap='Greys_r')"
      ],
      "metadata": {
        "id": "jt64hPRYPOWG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osvDQHOtvltZ"
      },
      "source": [
        "# Transfer learning\n",
        "\n",
        "Transfer learning is a method where a model is first trained on different, but similair, data and then fine-tuned on the real data. This means that the model does not need to start from scratch when training on the real data.\n",
        "\n",
        "In our case we need a dataset with something similair to hunting pits. Luckley for us there is a place with plenty of pits that we can use. The moon!\n",
        "\n",
        "NASA has digitized 1.3 milion lunar impact craters with a diameter with a diameter above 1-2 km. NASA also have a digital elevation model created by the Lunar Orbiter Laser Altimeter with a resolution of 118 m.\n",
        "\n",
        "Due to limitations in google colab we will not be able to train the lunar model from scratch. However we can pull the trained model used in the article from github.\n",
        "\n",
        "Start by downloading some lunar data to see if the pre-trained model works on impact craters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JKu1SCG_vpYP"
      },
      "outputs": [],
      "source": [
        "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/1gQ7dpNZLlWC5PR_J1MLJtmRpPfAkzNv5' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1gQ7dpNZLlWC5PR_J1MLJtmRpPfAkzNv5\" -O /content/moon.zip && rm -rf /tmp/cookies.txt\n",
        "!unzip /content/moon.zip -d /content/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWzZ8wnLPMll"
      },
      "source": [
        "## Inspect the lunar data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "njkNqEdMPPo6"
      },
      "outputs": [],
      "source": [
        "test_images_moon = tf.data.Dataset.load('/content/moon_small/test/') # These will be used to test the model to get a estimate of the preformance.\n",
        "test_examples_moon = list(test_images_moon.as_numpy_iterator())\n",
        "example1 = np.squeeze(test_examples_moon[0])\n",
        "image1 = example1[0,:,:,]\n",
        "label1 = example1[1,:,:,]\n",
        "\n",
        "example2 = np.squeeze(test_examples_moon[42])\n",
        "image2 = example2[0,:,:,]\n",
        "label2 = example2[1,:,:,]\n",
        "\n",
        "example3 = np.squeeze(test_examples_moon[70])\n",
        "image3 = example3[0,:,:,]\n",
        "label3 = example3[1,:,:,]\n",
        "\n",
        "# select and plot images\n",
        "plt.rcParams['figure.figsize'] = [12, 12]\n",
        "f, axarr = plt.subplots(3,2)\n",
        "axarr[0,0].set_title('Hillshade')\n",
        "axarr[0,0].imshow(image1, cmap='Greys_r')\n",
        "axarr[0,1].set_title('Labeled impact crater')\n",
        "axarr[0,1].imshow(label1,cmap='Greys')\n",
        "\n",
        "axarr[1,0].set_title('Hillshade')\n",
        "axarr[1,0].imshow(image2, cmap='Greys_r')\n",
        "axarr[1,1].set_title('Labeled impact crater')\n",
        "axarr[1,1].imshow(label2,cmap='Greys')\n",
        "\n",
        "axarr[2,0].set_title('Hillshade')\n",
        "axarr[2,0].imshow(image3, cmap='Greys_r')\n",
        "axarr[2,1].set_title('Labeled impact crater')\n",
        "axarr[2,1].imshow(label3,cmap='Greys')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pBZYAw4ebpYh"
      },
      "source": [
        "## Download the trained moon model and test it on some impact craters\n",
        "Start by cloning the github repository where the trained model is stored."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1iz2OQDdTUua"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/williamlidberg/Detection-of-hunting-pits-using-airborne-laser-scanning-and-deep-learning.git /content/code/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "icliBF8STm3T"
      },
      "source": [
        "Load the trained model and use it the moon"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YCpIyf6VbY5t"
      },
      "outputs": [],
      "source": [
        "import keras\n",
        "reconstructed_model = keras.models.load_model('/content/code/models/moon_hillshade.h5') # This is the pre-trained lunar model used in the paper.\n",
        "\n",
        "plt.rcParams['figure.figsize'] = [16, 16]\n",
        "f, axarr = plt.subplots(1,3)\n",
        "\n",
        "test_examples_moon = list(test_images_moon.as_numpy_iterator())\n",
        "test_example1 = np.squeeze(test_examples_moon[1])\n",
        "original_moon = test_example1[0,:,:,]\n",
        "label = test_example1[1,:,:,]\n",
        "\n",
        "testimage_moon = np.reshape(original_moon,[1,256,256,1])\n",
        "prediction = reconstructed_model.predict(testimage_moon)\n",
        "pred = np.squeeze(prediction)\n",
        "\n",
        "# Image\n",
        "axarr[0].set_title('Original image')\n",
        "axarr[0].imshow(original_moon, cmap='Greys_r')\n",
        "\n",
        "# Label\n",
        "axarr[1].set_title('Labeled pits')\n",
        "axarr[1].imshow(label, cmap='Greys_r')\n",
        "\n",
        "# Prediction\n",
        "axarr[2].set_title('Predicted craters')\n",
        "axarr[2].imshow(pred)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_EXdwVgYH-R"
      },
      "source": [
        "# Train the lunar model on data of hunting pits"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NeKr3z0jTw-s"
      },
      "outputs": [],
      "source": [
        "steps = len(train_images)/BATCH_SIZE\n",
        "reconstructed_model = keras.models.load_model('/content/code/models/moon_hillshade.h5')\n",
        "reconstructed_model.compile(tf.keras.optimizers.Adam(learning_rate=0.0001), loss=tf.keras.losses.BinaryFocalCrossentropy(gamma=2.0, from_logits=True), metrics=['acc', f1_m, recall_m])\n",
        "result_pretrained = reconstructed_model.fit(train_batches, validation_data=test_batches, epochs=10, steps_per_epoch=steps)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# summarize history for accuracy\n",
        "plt.rcParams['figure.figsize'] = [8, 8]\n",
        "plt.plot(result_pretrained.history['f1_m'])\n",
        "plt.title('Model accuracy using f1 score')\n",
        "plt.ylabel('f1 score')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')\n",
        "plt.xlim(0,10)\n",
        "plt.ylim(0,1)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "EhcVaaw_BV5C"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You could train the model for longer and get even better results but at some point the curve will level out. Test the model on the test data and compare the results to the original model."
      ],
      "metadata": {
        "id": "PExClhx6R2eV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UG0v4xUJcFYo"
      },
      "outputs": [],
      "source": [
        "reconstructed_model.evaluate(test_batches)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can take the new model and try it on a test image."
      ],
      "metadata": {
        "id": "dwFWJLOCSJ1s"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a99ItVQict9O"
      },
      "outputs": [],
      "source": [
        "plt.rcParams['figure.figsize'] = [16, 16]\n",
        "f, axarr = plt.subplots(1,4)\n",
        "\n",
        "test_examples = list(test_images.as_numpy_iterator())\n",
        "test_example1 = np.squeeze(test_examples[1])\n",
        "original = test_example1[0,:,:,]\n",
        "label = test_example1[1,:,:,]\n",
        "\n",
        "testimage = np.reshape(original,[1,256,256,1])\n",
        "# original model\n",
        "prediction = model.predict(testimage)\n",
        "pred = np.squeeze(prediction)\n",
        "\n",
        "#pre trained model\n",
        "prediction_pretrained = reconstructed_model.predict(testimage)\n",
        "pred_pretrained = np.squeeze(prediction_pretrained)\n",
        "\n",
        "# Image\n",
        "axarr[0].set_title('Original image')\n",
        "axarr[0].imshow(original, cmap='Greys_r')\n",
        "\n",
        "# Label\n",
        "axarr[1].set_title('Labeled pits')\n",
        "axarr[1].imshow(label, cmap='Greys_r')\n",
        "\n",
        "# Prediction from original model\n",
        "axarr[2].set_title('Predicted pits original model')\n",
        "axarr[2].imshow(pred>0.5, cmap='Greys_r')\n",
        "\n",
        "# Prediction\n",
        "axarr[3].set_title('Predicted pits pre-trained model')\n",
        "axarr[3].imshow(pred_pretrained>0.5, cmap='Greys_r')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAeYnacimdn2"
      },
      "source": [
        "## Questions\n",
        "\n",
        "\n",
        "\n",
        "1.   Will including more images produce better results?\n",
        "2.   Will an increased batch size improve the model?\n",
        "3.   Will the model preform better when training for more epochs?\n",
        "4.   Does training the original model for twice as lone result in the same accuracy as the pre-trained model?\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S-EJ5Ip1pw_2"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Reading\n",
        "\n",
        "https://arxiv.org/abs/1505.04597\n",
        "\n",
        "https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d\n",
        "\n",
        "https://naokishibuya.medium.com/up-sampling-with-transposed-convolution-9ae4f2df52d0\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEn_Gt_1p17R"
      },
      "source": [
        "## Contact\n",
        "William.lidberg@slu.se\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "mount_file_id": "1_Cj6vJ_0-pMR3zPPlrmqgqCF_QdT041U",
      "authorship_tag": "ABX9TyMhQLgWAxOq0Aj2bc1f1coZ",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}