Week5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 5\n",
    "\n",
    "Phew. You've handed in the assignment. But there's not resting now. We're just hitting out grove, so let's get going!! Much to get through today.\n",
    "\n",
    "## The plan for today\n",
    "\n",
    "We continue learning about dataviz by focusing on data with two variables and their relationships. The lecture today has the following parts:\n",
    "* In part 1, more lecturing on visualization and encodings.\n",
    "* In part 2, we talk about exploring data with two variables, make some logarithmic plots and think about what we have read in DAOST.\n",
    "* In part 3, we get a lightning intro to machine learning and the awesome package sklearn\n",
    "* And finally, in part 4, we use sklearn and have fun with linear regression.\n",
    "\n",
    "Ok. Now it's time to get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: More lecturing on dataviz\n",
    "\n",
    "So now we start learning more about the theory of visualization, digging into data encodings and representations.\n",
    "\n",
    "[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/zE6Nr8trdrw/0.jpg)](https://www.youtube.com/watch?v=zE6Nr8trdrw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Excercise 1:* Some questions about the video\n",
    ">\n",
    "> * Mention 10 examples of ways we can encode data.\n",
    "> * Are all encodings created equally? Why not? Can you think of an example from the previous lectures?\n",
    "> * Mention 3 encodings that are difficult for the human eye to parse. Can you find an example of a visualization online that uses one of those three?\n",
    "> * Explain in your own words: What is the problem with pie-charts?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Exploratory data visualzation, two variables  \n",
    "\n",
    "I told you how I love the Data Analysis with Open Source Tools book. You had to read Chapter 3, which is about visualizing data with two variables, before coming to class today. If you haven't yet, this is the time to do it! \n",
    "\n",
    "*Reading*: DAOST Chapter 3 up to *Graphical Analysis and Presentation Graphics* on page 68 in the PDF. **You will have to go and get it on DTU Learn due to the copyright stuff**.\n",
    "\n",
    "And now a few exercises to reflect on the text:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise 2.1:* Questions from DAOST Chapter 3.\n",
    "\n",
    "> * Looking at Fig 3-1, Janert writes \"the data itself shows clearly that the amount of random noise in the data is small\". What do you think his argument is?\n",
    "> * Can you think of a real-world example of a multivariate relationship like the one in Fig 3-3 (lower right panel)?\n",
    "> * What are the two methods Janert metions for smoothing noisy data? Can you think of other ones?\n",
    "> * What are residuals? Why is it a good idea to plot the residuals of your fit?\n",
    "> * Explain in your own words the point of the smooth tube in figure 3-7.\n",
    "> * What the h#ll is banking and what part of our visual system does it use to help us see patterns? What are potential problems with banking?\n",
    "> * Summarize the discussion of Graphical Analysis and Presentation Graphics on pp. 68-69 in your own words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Let's briefly talk about logarithms and logarithimic plots (if you take my networks class in the fall semester you'll se lots of loglog plots since they're important for understanding a key property of networks).\n",
    "\n",
    "*Exercise 2.2*: Logarithms and logarithmic plots. \n",
    "\n",
    "> * First, a couple of questions:\n",
    ">    * What kind of relationships will a semi-log plot help you discover?\n",
    ">    * What kind of functions will loglog plots help you see?\n",
    "> * Second, we are going to create a version of [this plot](https://github.com/suneman/socialdata2022/blob/main/files/CrimeOccurrencesByCategory.png) from Week 1, where you display the $y$-axis on log-scale. \n",
    "> * Third, let's also try a loglog plot. Inspired by [this article](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0183110) I would expect that especially theft could be characterized by a power law distribution also in San Francisco. Let's see if I'm right. \n",
    ">   - *Step 1:* Divide San Francisco into a grid **roughly** $100m \\times 100m$. You can, for example use numpy to do this, I would call `np.histogram2d`, and searching the internet, it seems that there are also [ways to do this in pandas](https://stackoverflow.com/questions/39254704/pandas-group-bins-of-data-per-longitude-latitude). The earth isn't flat,so `lat,` `lon` aren't really squares, but it is OK to ignore. \n",
    ">       * **Hint 1**. I really mean approximately 100 meters. It can also be 200 meters. Or 80 meters. Or 300.\n",
    ">       * **Hint 2**. Ignore outliers. We only want points that are on the SF peninsula\n",
    ">       * **Hint 3**. We've made a little example of how you can do the binning. Get it [here](https://github.com/suneman/socialdata2022/blob/main/lectures/Week5_binning.ipynb).\n",
    ">   - *Step 2:* Count the number of thefts occurring within each grid-square (use all data for all time).\n",
    ">   - *Step 3:* Tally the counts. Count the number of squares with $k=0$ thefts. We call this $N(0)$. Next, count the number of grids with one crime to get $N(k=1)$. Keep going like this all the way up to $k=C_{max}$, where $C_{max}$ is the highest count of crimes you find in any grid space. \n",
    ">     * *Extra tip*: If you want all the details on binning for loglog axes, you can check out [Lecture 2, Part 3](https://github.com/SocialComplexityLab/socialgraphs2021/blob/main/lectures/Week2.ipynb) in my social graphs course.\n",
    ">   - *Step 4:* Plot the distribution of $k+1$ vs $N(k)$ on linear axes.\n",
    ">   - *Step 5:* Plot the distribution of $k+1$ vs $N(k)$ on loglog axes.\n",
    ">   - *Step 6:* Answer the question. Was Sune correct in assuming that there is a power-law distribution of theft?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Lightning intro to Machine Learning and sklearn\n",
    "\n",
    "Before we get to linear regression and as we are going to use a bit of Machine Learning in the following lectures too, we are going to have a quick intro to Machine Learning.\n",
    "\n",
    "We kick off the machine-learning part by watching a video lecture on *What Machine Learning is*. The lecture (and the one in next weeks) have been prepared by our very own expert, Ole Winter, whose work focuses on Machine Learning. The lectures + slides have been prepared especially for you guys by Ole, and lovingly edited by Sune.\n",
    "\n",
    "**What is machine learning?**\n",
    "[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/SsCYF9tDY9Y/0.jpg)](https://www.youtube.com/watch?v=SsCYF9tDY9Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise 3.1* We now tie everything together (video, DAOST etc.) by answering a few questions:\n",
    "> \n",
    "> * What are the different categories of Machine Learning? Where does linear regression fall in?\n",
    "> * What is the difference between Machine Learning and Statistics? How does this idea connect to the purpose of linear regression according to Janert (DAOST Ch. 3)?\n",
    "> * I think figure 3-14 (DAOST Ch. 3) makes an important point about linear fits that is rarely made. What is it? \n",
    "> * What is problematic about using straight lines to fit the data in Fig 3-5? (Something similar is actually the topic of a *Nature* article from 2004 get it [here](https://github.com/suneman/socialdata2022/blob/main/files/regrunners.pdf). And an extra [link](http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3366/pdf/imm3366.pdf) on this topic for the students who know Danish)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, but how do we apply different machine learning models in practice? \n",
    "\n",
    "The amazing package `sklearn` is state-of-the-art machine learning for Python. It's used in companies big and small all over the world and in lots of academic papers.\n",
    "\n",
    "Today, we start with a high level overview presented in [this tutorial](https://scikit-learn.org/stable/tutorial/basic/tutorial.html). **Try it out**: Read/work through the first three sections (*Machine learning: the problem setting*, *Loading an example dataset*, *Learning and predicting*) to get a sense of data types and syntax.\n",
    "\n",
    "*Exercise 3.2*: Did you read the text?\n",
    ">\n",
    "> * Describe in your own words how data is organized in `sklearn` (how does a *dataset* work according to the tutorial)?\n",
    "> * What is the dimensionality of the `.data` part of a dataset and what is the size of each dimension?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Linear Regression\n",
    "\n",
    "So, now it's time for fun with standard linear regression! We'll get into that by asking the following question. \n",
    "\n",
    "> *Which pair of focus crimes have the the most similar temporal pattern across the week? (And which pair is most dissimilar).*\n",
    "\n",
    "Below I list the focus-crimes for your convenience"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "focuscrimes = set(['WEAPON LAWS', 'PROSTITUTION', 'DRIVING UNDER THE INFLUENCE', 'ROBBERY', 'BURGLARY', 'ASSAULT', 'DRUNKENNESS', 'DRUG/NARCOTIC', 'TRESPASS', 'LARCENY/THEFT', 'VANDALISM', 'VEHICLE THEFT', 'STOLEN PROPERTY', 'DISORDERLY CONDUCT'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to answer this question is to plot the activity for all pairs of crimetypes as scatter plot per pair. One crime type on each axis, and where each point in the scatter corresponds to an hour of the week, and the number of crimes of crime-type 1 is on the $x$-axis and the number of crimes of crime-type 2 is on the $y$-axis. (So there will be 168 points in each scatterplot.) If we look at 14 focus crimes that results in 91 pairwise comparisons. \n",
    "\n",
    "*Exercise 4.1*: Create the 91 scatterplots.\n",
    "> * Display the plots in a $7$ by $13$ subplot matrix. You can use matplotlib's `subplot` to organize those plots. With $7$ across and $13$ down, you should be able to squeeze them all onto a single [a4](https://en.wikipedia.org/wiki/ISO_216#A_series) page.\n",
    ">     * Make sure to label each one with the two crime-types you're comparing so we can easily inspect visually.\n",
    ">     * Make sure that that you squeeze the subplots closely together so each plot can be as big as possible. \n",
    "> * Just inspecting this matrix, which crime-types look correlated and which one look like they're very different?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Next it's time for the linear regression. Janert writes about this on page 63-66. \n",
    "\n",
    "There is a closed-form solution for linear regression. If you want to find the best straight line $y = ax + b$ fit to a set of $N$ points $\\{(x_1,y_1), (x_2,y_2), \\ldots (x_N,y_N)\\}$, the value of $b$ is\n",
    "\n",
    "$$\n",
    "\\tag{1}\n",
    "b = \\langle y \\rangle - a \\langle x \\rangle,\n",
    "$$\n",
    "\n",
    "where $\\langle x \\rangle = (1/N)\\sum_i x_i$ is the mean value of the $x_i$ and $\\langle y \\rangle = (1/N)\\sum_i x_i$ is the mean value of the $y_i$. \n",
    "\n",
    "And the value for the slope $a$ is \n",
    "\n",
    "$$\n",
    "\\tag{2}\n",
    "a = \\frac{\\sum_{i=1}^N \\left( x_iy_i \\right) - N \\langle x\\rangle\\langle y\\rangle }{\\sum_{i=1}^N\\left( x_i^2 \\right) - N\\langle x\\rangle^2}.\n",
    "$$\n",
    "\n",
    "\n",
    "A couple of years ago, Sune actually derived the whole thing. I've taken it out of the notebook. But if you'd like to take a look (it's a fun and instructive little exercise), you can find it **[here](https://github.com/suneman/socialdata2021/blob/main/lectures/LinearRegressionDerived.ipynb)**. \n",
    "\n",
    "We are going to focus on the fit for now, but keep in mind what we have learnt so far about the purpose of linear regression! \n",
    "\n",
    "*Exercise 4.2:* Linear regression using sklearn.\n",
    ">\n",
    "> * Using [`LinearRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) from `sklearn`, compute the slope and intercept for each couple of focus-crimes (from Ex. 4.1) and add a linear fit to each of the 91 scatterplots above.\n",
    "> * **(Optional but highly recommended)** Using the formulas we derived above (Equation 1 and 2), calculate the slopes for $a$ and $b$ in each case. Compare the results with the ones obtained with sklearn to check that everything is working as expected.\n",
    "> * **(Optional)** You can add even more information to this plot by coloring each point according to its hour of the week. So create a gradient going from one color to another, and color each point according to the gradient. (So let's say your two colors are red and blue, then the Sunday, midnight to 1am bin will be red and the following Sunday, 11pm - midnight bin will be blue)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "But the question we started with was \"Which pair of focus crimes have the the most similar temporal pattern across the week?\". We haven't really answered that yet. So let's calculate one last thing: $R^2$. You probably also remember this one. \n",
    "\n",
    "Basically $R^2$ is a measures of how good a linear fit is. You can [read about $R^2$ on wikipedia](https://en.wikipedia.org/wiki/Coefficient_of_determination). \n",
    "\n",
    "*Exercise 4.3:* Goodness of fit as a measure of correlation. \n",
    "> * Write a little function to calculate $R^2$ alongside each linear fit, and compare it with the one provided by the function `score()` in `sklearn`.\n",
    "> * **According to the fits and associated $R^2$**, which pair of crimes have the **most similar** temporal pattern. Discuss your finding: Does it make sense? Why?/Why not?\n",
    "> * According to your fits and associated measure of $R^2$, which pair of crimes have the **most dissimilar** temporal pattern. Discuss your finding: Does it make sense? Why?/Why not?\n",
    "> * Explain the connection between $R^2$ and the [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}