03_dask.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "99a571e0-4b6e-4bee-9934-ec57ef3dd3e5",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <img src = \"./assets/dask-logo.svg\" alt=\"Dask logo\" width=\"20%\">\n",
    "</div>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "921d5687-c9bb-41f2-aa24-656969dfbe37",
   "metadata": {},
   "source": [
    "# Dask for big data computations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9b7420-dc58-4ac5-ac02-5355e36309b3",
   "metadata": {},
   "source": [
    "## 💎 Key takeaways\n",
    "\n",
    "By the end of this notebook, you will:\n",
    "- **Have a basic understanding of Dask and Dask-Gateway and,**\n",
    "- **Know how to launch a Dask cluster on Nebari.**\n",
    "\n",
    "Integrating Dask-Gateway on to a base JupyterHub is a non-trivial task, as such Dask-Gateway comes pre-configured on Nebari for you!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a914b445-abdd-4c68-a2b4-396a389282e8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## What is [Dask](https://www.dask.org/)?\n",
    "\n",
    "> Dask is a parallel and distributed computing library that scales the existing Python and PyData ecosystem.\n",
    ">\n",
    "> Dask can scale up to your full laptop capacity and out to a cloud cluster.\n",
    ">\n",
    "> *~ Source: [Dask Tutorial](https://tutorial.dask.org/00_overview.html)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59e71afd-4537-49b6-a5f3-5eb97295db44",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"assets/dask-example-dashboard.gif\" alt=\"Dask computation with task stream annd progress bar\" width=\"70%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6721517f-72fa-4a78-810a-ae2c551ea375",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "As a brief overview, Dask offers:\n",
    "\n",
    "<details>\n",
    "    <summary><b>Dask collections:</b> Drop-in replacements for Numpy, Pandas, and other Python packages that make it easy to parallelize your code</summary>\n",
    "\n",
    "**Dask Collections** (or APIs) are what you as a user will interact with. They create the logic of how your computation will be executed in a multi-core and distributed fashion, on larger-than-memory datasets. If your dataset is too big for your single machine, you can use Dask to still work with the data as you would if you were using NumPy or pandas. \n",
    "\n",
    "The high-level APIs include [Dask Array](https://docs.dask.org/en/stable/array.html), a subset of NumPy's `ndarray` interface, [Dask Bag](https://docs.dask.org/en/stable/bag.html), an implementation of generic Python operations like `map`, `filter`, `groupby`, and [Dask DataFrame](https://docs.dask.org/en/stable/dataframe.html), a subset of pandas `DataFrame` and `Series` interface. \n",
    "    \n",
    "The advantage of the approach taken by Dask is that if you are familiar with NumPy's `ndarray` or pandas `DataFrame`, you can get started using Dask Array or Dask DataFrame rather quickly because they follow a similar syntax.\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "    <summary><b>Dask cluster:</b> A collection of dask workers controlled by a central dask scheduler.</summary>\n",
    "\n",
    "    The **Dask cluster** consists of a (local or distributed) scheduler and worker machines. It represent the part of the library dedicated to actually administering and executing your computations on a distributed cluster of machines.\n",
    "</details>\n",
    "    \n",
    "<details>\n",
    "    <summary><b>Additional tools:</b></summary>\n",
    "    - Dask-ML\n",
    "    - Dask-sql\n",
    "    - ...\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "735d78af-b6c1-444b-a2ab-d0713f91e05f",
   "metadata": {},
   "source": [
    "## What is [Dask Gateway](https://gateway.dask.org/)?\n",
    "\n",
    "Dask can be setup to run on a variety of backend clusters, including Kubernetes, Docker, HPC, YARN/Hadoop, and more.\n",
    "\n",
    "> **Dask Gateway** allows you to launch and use Dask clusters in a shared, centrally managed cluster environment, without requiring users to have direct access to the underlying cluster backend (e.g.. Kubernetes, Hadoop/YARN, HPC Job queues, etc.)\n",
    ">\n",
    "> <img src=\"./assets/dask-gateway-overview.png\" alt-text=\"Diagram of Dask-Gateway architecture\" width=\"50%\"></img>\n",
    "> \n",
    "> *~ Source: [gateway.dask.org](https://gateway.dask.org/)*\n",
    "\n",
    "For the purposes of this Nebari tutorial, when we refer to Dask running on a distributed cluster, we mean connecting to Dask Gateway.\n",
    "\n",
    "This means that users with access to Dask Gateway (more on user permissions in a later notebook) need to connect to the gateway to submit their workloads to the Dask cluster. For a concrete example of how this done, check out the links at the bottom of this page."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2097b48f-4607-49d0-a118-923867f60813",
   "metadata": {},
   "source": [
    "## 👉 Your turn:\n",
    "\n",
    "Here we run through a basic example of how to use Dask Gateway in Nebari: [finance_examples/03a_dask_gateway_adaptive_scaling.ipynb](./finance_examples/03a_dask_gateway_adaptive_scaling.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54ebfdd0-9cff-47f8-a42d-0adf1a3b6815",
   "metadata": {},
   "source": [
    "## Why do we include Dask and Dask Gateway with Nebari?\n",
    "\n",
    "* In PyData, Dask has become foundational for out-of-memory computation and lots of Nebari uses also use Dask.\n",
    "* Dask's features like adaptive scaling and diagnostic dashboards can help you manage your big data computation and costs.\n",
    "* Making sure Dask deployments work on your cloud platform is non-trivial, so we ship it built-in to make your workflows more efficient."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10a8ca5b-2c5c-41a7-a157-5a4e2d338a49",
   "metadata": {},
   "source": [
    "---\n",
    "## 👏 Next:\n",
    "* [04_visualizations_and_dashboards](./04_visualizations_and_dashboards.ipynb)\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21858540-4ff8-4780-8d7e-8c3bedfa137d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "global-global-nebari-demo",
   "language": "python",
   "name": "conda-env-global-global-nebari-demo-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}