Lesson 2 - Introduction to Docker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<img src='https://regmedia.co.uk/2015/09/29/docker_logo.jpg'/>\n",
    "\n",
    "# Introduction to Docker\n",
    "\n",
    "Docker is a way to package apps, and all of their dependencies, into **images** that can be easily loaded onto any operating system and executed (\"instantiated\") as a **container**\n",
    "\n",
    "Docker containers are like a virtual machine (like the one you are using to take this course), but they are much much smaller because the image used to instantiate a container doesn't contain an operating system.  A container contains only the app, and the other things the app needs to run - all other functionality comes from the host operating system.  Apps are \"installed\" and configured inside of the container so that the user doesn't need to go through the installation or configuration process - the app \"just works\".\n",
    "\n",
    "As data scientists, you are going to create a lot of software in your lives!  Docker (and other associated projects) is a good option for making your software available for others to use.\n",
    "\n",
    "In this introductory course, we will only learn how to USE Docker images/containers, we will not learn how to CREATE them (this may be added to the Bioinformatics Programming Challenges course this year!).\n",
    "\n",
    "Surf to:  https://hub.docker.com/explore/\n",
    "\n",
    "This page lets you browse or search for images that have apps you want.  For example, if you wanted to write some code in Python, but didn't have Python installed, you could download the Python docker image (search for it).  Similarly, if you wanted to run the BLAST Nucleotide/amino acid sequence search program, you could use it's Docker distribution instead of installing it (search for Blast.... search for \"bioinformatics\")\n",
    "\n",
    "## Installing Docker\n",
    "\n",
    "Docker is, itself, an app, so it must be installed.  **I have done this for you already**, but here are some recent instructions for how to install it on Linux:\n",
    "\n",
    "#### get the keys to validate the docker repository\n",
    "\n",
    "    sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 \\\n",
    "      --recv-keys 58118E89F3A912897C070ADBF76221572C52609D\n",
    " \n",
    "#### Add the official docker repository to your apt\n",
    " \n",
    "    sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'\n",
    " \n",
    "#### Update \n",
    " \n",
    "    sudo apt update\n",
    "\n",
    "#### Install some Docker dependencies\n",
    " \n",
    "    sudo apt install linux-image-generic linux-image-extra-virtual\n",
    " \n",
    "#### Reboot\n",
    " \n",
    "    sudo reboot\n",
    " \n",
    "    sudo apt install docker-engine\n",
    " \n",
    "#### add your user id to the docker group so that permissions are not a problem\n",
    "\n",
    "    sudo usermod -a -G docker osboxes\n",
    "    \n",
    "\n",
    "## Getting Docker Images\n",
    "\n",
    "To \"get\" an image, the easiest thing to do is to \"pull\" it.  **pull** will only download the Image to your system, it will not run it (i.e. it will not instantiate it into a Container).  The **pull** step **is not strictly necessary** - the **docker run** command (below) will also do a **pull** before it attempts to run the Image; however, if you wanted to use an existing Image as the base for creating your own software app and your own Docker Image, you might want to **pull** it without running it.\n",
    "\n",
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using default tag: latest\n",
      "latest: Pulling from library/hello-world\n",
      "\n",
      "\u001b[1BDigest: sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9\n",
      "Status: Downloaded newer image for hello-world:latest\n",
      "docker.io/library/hello-world:latest\n"
     ]
    }
   ],
   "source": [
    "docker pull hello-world\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Docker Containers\n",
    "\n",
    "The easiest way to run an image - i.e. create a Container from that Image - is to simply **run** it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Hello from Docker!\n",
      "This message shows that your installation appears to be working correctly.\n",
      "\n",
      "To generate this message, Docker took the following steps:\n",
      " 1. The Docker client contacted the Docker daemon.\n",
      " 2. The Docker daemon pulled the \"hello-world\" image from the Docker Hub.\n",
      "    (amd64)\n",
      " 3. The Docker daemon created a new container from that image which runs the\n",
      "    executable that produces the output you are currently reading.\n",
      " 4. The Docker daemon streamed that output to the Docker client, which sent it\n",
      "    to your terminal.\n",
      "\n",
      "To try something more ambitious, you can run an Ubuntu container with:\n",
      " $ docker run -it ubuntu bash\n",
      "\n",
      "Share images, automate workflows, and more with a free Docker ID:\n",
      " https://hub.docker.com/\n",
      "\n",
      "For more examples and ideas, visit:\n",
      " https://docs.docker.com/get-started/\n",
      "\n"
     ]
    }
   ],
   "source": [
    "docker run hello-world\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "To see what Images you have, use the **docker images** command\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE\n",
      "hello-world         latest              bf756fb1ae65        5 months ago        13.3kB\n"
     ]
    }
   ],
   "source": [
    "docker images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at how small hello-world is!  This is why Docker is becoming more popular than Virtual Machines, or even software installations (I don't have to **install** the hello-world app, I simply **docker run** it!)\n",
    "\n",
    "Once you are finished with an Image, you can delete it with the **docker rmi** (remove image) command.  Note that you must use the IMAGE ID of the docker Image you want to remove - in this case, the hello-world Image.  However, because we just ran it, we created a Container.  We must first stop the Container.  See the error message when I try to remove the hello-world Image...:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error response from daemon: conflict: unable to delete bf756fb1ae65 (must be forced) - image is being used by stopped container 80249f02179c\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "docker rmi bf756fb1ae65   # this will be different every time...\n",
    "# note that the error below tells us the container id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "80249f02179c\n"
     ]
    }
   ],
   "source": [
    "docker rm 80249f02179c   # this will be different every time  (note that it is \"rm\" (remove) not \"rmi\" (remove image))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Untagged: hello-world:latest\n",
      "Untagged: hello-world@sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9\n",
      "Deleted: sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b\n",
      "Deleted: sha256:9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63\n"
     ]
    }
   ],
   "source": [
    "docker rmi bf756fb1ae65  # now that I have removed the Container, I can remove the Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE\n"
     ]
    }
   ],
   "source": [
    "docker images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "## Running more complex software\n",
    "\n",
    "hello-world is not very complex :-)  Docker containers can contain many kinds of functionalities, from web servers to databases to entire operating systems.  Often you will need to configure the app(s) inside of the Container, or give the Container access to certain folders on your computer, or open certain ports into the Container (e.g. so that the world can access your Web server)\n",
    "\n",
    "This is achieved using **docker run**, with additional arguments.\n",
    "\n",
    "for example, to \"map\" a communication port from the host computer to the Container (e.g. port 80 is the normal port for a Web server), you would add:\n",
    "\n",
    "     -p 80:80    # this means, the host port 80 goes to the Container port 80\n",
    "     \n",
    "You can also pass Environment Variables using the '-e' argument\n",
    "\n",
    "     -e MY_VARIABLE_HERE=some_value\n",
    "\n",
    "Every Image will have different instructions about how to run it, [so I wont (can't!) explain all of the options here](https://docs.docker.com/engine/reference/commandline/run/#options).  Just read the instructions for the Image you want to use.\n",
    "\n",
    "The last argument we will learn is \"--name\".  This is used to give a specific (memorable) name to your Container.  Moreover, it allows you to use the same Image, with different configurations, in different Containers (e.g. you run the **same Web Server Image twice**, one Container runs your Personal Website (--name my-personal-website) and the other one runs your Laboratory Website (--name lab-web-server).  The two Containers can run at the same time! \n",
    "\n",
    "\n",
    "## The MySQL Database Server Docker Image\n",
    "\n",
    "MySQL is a very popular, open-source database system widely used in Bioinformatics.  MySQL can be installed directly onto your computer; however, it can sometimes be a bit difficult to configure it.  Using the Docker Image, it becomes extremely easy!\n",
    "\n",
    "https://hub.docker.com/search/?isAutomated=0&isOfficial=0&page=1&pullCount=0&q=mysql&starCount=0\n",
    "\n",
    "The \"Official MySQL\" Image is the one we want.  Click on it now.\n",
    "\n",
    "https://hub.docker.com/_/mysql/\n",
    "\n",
    "### Image tags\n",
    "\n",
    "There are often many versions of a Docker image.  If you don't care which version you use, there is a tag called \"latest\" which will give you the latest... **BUT YOU SHOULD ALWAYS CARE!!!**  You cannot do reproducible science if you don't know what version of software you are using, or when the software you are using suddenly changes in the middle of your PhD!\n",
    "\n",
    "These are the current MySQL Docker Images.\n",
    "\n",
    "    Supported tags and respective Dockerfile links\n",
    "\n",
    "    8.0.11, 8.0, 8, latest (8.0/Dockerfile)\n",
    "    5.7.22, 5.7, 5 (5.7/Dockerfile)\n",
    "    5.6.40, 5.6 (5.6/Dockerfile)\n",
    "    5.5.60, 5.5 (5.5/Dockerfile)\n",
    "    \n",
    "\n",
    "I know that Version 8.x.x has some problems, so I don't want to use it.  I will use the last version from the v5 series (5.7.22).  The way to refer to a specific version of an Image is to say:   imagename:tag\n",
    "\n",
    "For example, the Image we want is:\n",
    "\n",
    "       mysql:5.7.22\n",
    "\n",
    "\n",
    "You can read the rest of the instructions (lower in the page) by yourself.  Basically, the main argument you need to provide the Container is the password you want to use for the 'root' database user (MYSQL_ROOT_PASSWORD=some_password_here)\n",
    "\n",
    "One instruction that they (annoyingly!) forget to tell you is that, to access the mysql server inside of the Container, you need to attach port 3306 to the Container port 3306.\n",
    "\n",
    "The final command line to start the MySQL Server is:\n",
    "\n",
    "     docker run --name course-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=root -d mysql:5.7.22\n",
    "    \n",
    "this runs mysql v5.7.22, with the root password of 'root', with port connections of 3306->3306, and creates a Container named \"course-mysql\".  It also \"detaches\" (-d), meaning that after you execute the **docker run** command, the Container will continue to run even if you close the terminal window (it will NOT continue to run after you shut-down the computer!)\n",
    "\n",
    "Open a terminal window and execute that command now.... then come back to this page\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "To see what Containers are running use the **docker ps** command:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES\n",
      "90455ce532a5        mysql:5.7.22        \"docker-entrypoint.s…\"   7 minutes ago       Up 7 minutes        0.0.0.0:3306->3306/tcp   course-mysql\n"
     ]
    }
   ],
   "source": [
    "docker ps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that your container is named \"course-mysql\".  You can now use this container name to quickly start or stop the MySQL server using the **docker stop** and **docker start** commands:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "course-mysql\n"
     ]
    }
   ],
   "source": [
    "docker stop course-mysql"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES\n"
     ]
    }
   ],
   "source": [
    "docker ps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "course-mysql\n"
     ]
    }
   ],
   "source": [
    "docker start course-mysql"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES\n",
      "90455ce532a5        mysql:5.7.22        \"docker-entrypoint.s…\"   7 minutes ago       Up 3 seconds        0.0.0.0:3306->3306/tcp   course-mysql\n"
     ]
    }
   ],
   "source": [
    "docker ps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE AGAIN** when you **docker start** it starts the Container __with the same configuration that was used to create it__.  This is very useful when you have various configurations of the same software."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "\n",
    "\n",
    "</pre>\n",
    "\n",
    "#  This is all we will learn for now.\n",
    "\n",
    "But there is a lot more to learn about Docker!  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}