SEN_Transport_agriculture-prep_2021-12-22.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network analysis in Senegal\n",
    "\n",
    "### Objectives\n",
    "    1)\tUse measures of road-based accessibility to identify road segments that, if rehabilitated, would improve agricultural market activities in Senegal, including during flood conditions.\n",
    "    2)\tGain a better understanding of the accessibility, connectivity, and criticality of roads in Senegal in relationship to agricultural origins, processing & transfer sites, and markets.\n",
    "\n",
    "To this end, the team will develop an accessibility model which measures the travel time from sites of agricultural production to their nearest populated areas, processing centers, and markets. \n",
    "\n",
    "### Datasets for analysis\n",
    "#### ORIGIN\n",
    "    1) agriculture: MapSPAM 2017. Measuring value in international dollars.\n",
    "    2) agriculture: UMD Land Cover 2019 30m. Assign MapSPAM value onto land cover cropland class for more precise origin information.\n",
    "    3) population: WorldPop 2020, UN-adjusted.\n",
    "    4) settlement extent: GRID3 2020.\n",
    "#### DESTINATION\n",
    "    4) markets: derived from WorldPop 2020 and GRID3 2020 urban clusters.\n",
    "    5) agricultural processing hubs: to be acquired.\n",
    "#### TRAVEL ROUTE\n",
    "    6) roads: OpenStreetMap, July 2021.\n",
    "    7) elevation: \n",
    "#### OBSTACLE\n",
    "    8) flood: FATHOM. 1-in-10, 20, and 50 year flood return periods. \n",
    "#### INTERVENTION\n",
    "    9) upcoming road projects: AGEROUTE interventions separate from the World Bank-financed project\n",
    "    10) targeted road projects: critical road segments identified by this accessibility model's baseline outputs\n",
    "\n",
    "\n",
    "### Model design\n",
    "#### Basic formula: \n",
    "    (a) Off-road driving time from origin to closest road node\n",
    "    +\n",
    "    (b) Driving time from road node in (a) to a destination (closeness measured by road segments speeds)\n",
    "\n",
    "#### Model origin & destination (OD) sets:\n",
    "    A)\tTravel time from an area that has agricultural value/potential to the nearest processing hub (if provided).\n",
    "    B)\tTravel time from an area that has agricultural value/potential to the nearest larger settlement, (“larger” settlement identified using a case-appropriate population metric to be determined).\n",
    "    C)\tTravel time from an area that has agricultural value/potential to the nearest market.\n",
    "    D)\tTravel time from all settlements to the nearest market.\n",
    "    E)\tTravel time from larger settlements to the nearest market.\n",
    "\n",
    "#### Before/after scenarios for each OD set:\n",
    "    1)\tPre-project, baseline weather: No inclement weather. Road network status as of November 2021.\n",
    "    2)\tPre-project, flood: 1-in-10, 1-in-20 and 1-in-50 year flood return period. Road network status as of November 2021.\n",
    "    3)\tPost-project, baseline weather: No inclement weather. Road network status if X number of critical road segments to high-value areas are protected (i.e., their travel times reduced).\n",
    "    4)\tPost-project, flood: 1-in-10 year flood return period. Road network status if X number of critical road segments to high-value areas are protected (i.e., their travel times reduced).\n",
    "\n",
    "#### Notes:\n",
    "    --Destinations are expected to be proximal to the road network, so no measure is taken between road and destination.\n",
    "    --All travel times will be assigned to each model variation’s point of origin; the aggregation up to admin areas is possible if desired.\n",
    "    --Obstacles & interventions modify the road segment speeds. Basic formula is then applied to the modified road network.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prep workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\wb527163\\\\GEO-Cdrive-Grace'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os, sys\n",
    "GISFolder = os.getcwd()\n",
    "GISFolder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: needed to reinstall rtree due to geopandas import error. Did so in the console. \n",
    "# conda install -c conda-forge rtree=0.9.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load and filter osm network (step 1)\n",
    "import geopandas as gpd\n",
    "from geopandas import GeoDataFrame\n",
    "import pandas as pd\n",
    "import time\n",
    "sys.path.append(r\"C:\\Users\\wb527163\\.conda\\envs\\geo\\GOSTnets-master\")\n",
    "import GOSTnets as gn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import osmnx as ox\n",
    "import numpy as np\n",
    "import rasterio as rt\n",
    "import shapely\n",
    "from shapely.geometry import Point, box\n",
    "from shapely.ops import unary_union\n",
    "from shapely.wkt import loads\n",
    "from shapely import wkt\n",
    "from shapely.geometry import LineString, MultiLineString, Point\n",
    "import peartree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Might not use these\n",
    "import fiona\n",
    "from osgeo import gdal\n",
    "import importlib\n",
    "import matplotlib.pyplot as plt\n",
    "import subprocess, glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\wb527163\\\\GEO-Cdrive-Grace\\\\SEN-Cdrive'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pth = os.path.join(GISFolder, \"SEN-Cdrive\") # Personal folder system for running model.\n",
    "pth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\wb527163\\\\GEO-Cdrive-Grace\\\\SEN-Cdrive\\\\outputs'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out_pth = os.path.join(GISFolder, \"SEN-Cdrive\\outputs\") # For storing intermediate outputs from the model.\n",
    "out_pth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'R:\\\\SEN\\\\GEO'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "team_pth = 'R:\\\\SEN\\\\GEO' # This is where the unmodified input data is stored. Finalized outputs also housed here.\n",
    "team_pth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare and clean the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notes:\n",
    "# OSM road network is in WGS84. Projected each dataset to match.\n",
    "# Starting as CSV (dataframe) and deriving geometry from there tends to avoid read errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Previously: joined MapSPAM agricultural values (\"spam\") to the much higher resolution but empty cropland class of UMD landcover. Divided the ag value across all cropland landcover points (\"LC\") located within a spam cell.\n",
    "\n",
    "#### Now, dealing with spatial gaps so that all MapSPAM values have a point of origin for the graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pseudocode:\n",
    "    4. agriculture.shp = Spatial join SPAM[\"val\"] and SPAM[\"SID\"] onto lc.shp (full outer join)\n",
    "    5. Where [\"SID\"] = NULL, [\"val\"] = 0\n",
    "    6. Where [\"lc_crop\"] = NULL:\n",
    "        SPAM_centroid.shp = SPAM.shp to centroid (retain [\"val\"] field)\n",
    "        SPAM_cent_subset.shp = Select SPAM_centroid based on lc_crop = NULL and export\n",
    "        Append agriculture.shp with SPAM_cent_subset.shp\n",
    "    7. [\"lc_sum\"] = Sum [\"lcID\"] per [\"SID\"]\n",
    "    8. Divide SPAM value by lc_sum and assign it to landcover[\"spam_V\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 2263 entries, 0 to 2262\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count  Dtype   \n",
      "---  ------        --------------  -----   \n",
      " 0   Id            2263 non-null   int64   \n",
      " 1   gridcode      2263 non-null   int64   \n",
      " 2   Shape_Length  2263 non-null   float64 \n",
      " 3   Shape_Area    2263 non-null   float64 \n",
      " 4   geometry      2263 non-null   geometry\n",
      "dtypes: float64(2), geometry(1), int64(2)\n",
      "memory usage: 88.5 KB\n"
     ]
    }
   ],
   "source": [
    "spam = gpd.read_file(r\"C:\\Users\\wb527163\\GEO-Cdrive-Grace\\SEN-Cdrive\\scratch.gdb\", layer=\"spam_poly_clip\")\n",
    "spam.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>geometry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1387</td>\n",
       "      <td>50698</td>\n",
       "      <td>MULTIPOLYGON (((-12.91734 13.75031, -13.00067 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2448</td>\n",
       "      <td>320005</td>\n",
       "      <td>MULTIPOLYGON (((-11.91734 12.08364, -12.00067 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>365</td>\n",
       "      <td>4503</td>\n",
       "      <td>MULTIPOLYGON (((-13.50067 15.41696, -13.50067 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2150</td>\n",
       "      <td>152650</td>\n",
       "      <td>MULTIPOLYGON (((-12.83400 12.50031, -12.91734 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1961</td>\n",
       "      <td>105340</td>\n",
       "      <td>MULTIPOLYGON (((-16.50065 12.75031, -16.58399 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2258</th>\n",
       "      <td>695</td>\n",
       "      <td>468582</td>\n",
       "      <td>MULTIPOLYGON (((-12.33400 14.75030, -12.41734 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2259</th>\n",
       "      <td>1431</td>\n",
       "      <td>362959</td>\n",
       "      <td>MULTIPOLYGON (((-14.08400 13.66697, -14.16733 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2260</th>\n",
       "      <td>1319</td>\n",
       "      <td>232790</td>\n",
       "      <td>MULTIPOLYGON (((-14.00066 13.83364, -14.08400 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2261</th>\n",
       "      <td>905</td>\n",
       "      <td>1133888</td>\n",
       "      <td>MULTIPOLYGON (((-15.58399 14.41697, -15.66732 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2262</th>\n",
       "      <td>42</td>\n",
       "      <td>3259</td>\n",
       "      <td>MULTIPOLYGON (((-14.16733 16.58363, -14.25066 ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2263 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      ID_spam  grid_val                                           geometry\n",
       "0        1387     50698  MULTIPOLYGON (((-12.91734 13.75031, -13.00067 ...\n",
       "1        2448    320005  MULTIPOLYGON (((-11.91734 12.08364, -12.00067 ...\n",
       "2         365      4503  MULTIPOLYGON (((-13.50067 15.41696, -13.50067 ...\n",
       "3        2150    152650  MULTIPOLYGON (((-12.83400 12.50031, -12.91734 ...\n",
       "4        1961    105340  MULTIPOLYGON (((-16.50065 12.75031, -16.58399 ...\n",
       "...       ...       ...                                                ...\n",
       "2258      695    468582  MULTIPOLYGON (((-12.33400 14.75030, -12.41734 ...\n",
       "2259     1431    362959  MULTIPOLYGON (((-14.08400 13.66697, -14.16733 ...\n",
       "2260     1319    232790  MULTIPOLYGON (((-14.00066 13.83364, -14.08400 ...\n",
       "2261      905   1133888  MULTIPOLYGON (((-15.58399 14.41697, -15.66732 ...\n",
       "2262       42      3259  MULTIPOLYGON (((-14.16733 16.58363, -14.25066 ...\n",
       "\n",
       "[2263 rows x 3 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spam.rename(columns={'Id':'ID_spam', 'gridcode':'grid_val'}, inplace=True)\n",
    "spam = spam.drop(columns=['Shape_Length', 'Shape_Area'])\n",
    "spam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13941001 entries, 0 to 13941000\n",
      "Data columns (total 5 columns):\n",
      " #   Column    Dtype   \n",
      "---  ------    -----   \n",
      " 0   ID_spam   float64 \n",
      " 1   grid_val  float64 \n",
      " 2   ID_LC     float64 \n",
      " 3   val       float64 \n",
      " 4   geometry  geometry\n",
      "dtypes: float64(4), geometry(1)\n",
      "memory usage: 531.8 MB\n"
     ]
    }
   ],
   "source": [
    "agriculture = gpd.read_file(r\"C:\\Users\\wb527163\\GEO-Cdrive-Grace\\SEN-Cdrive\\scratch.gdb\", layer=\"agriculture\")\n",
    "agriculture.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spam = spam.to_crs(\"EPSG:31028\")\n",
    "agriculture = agriculture.to_crs(\"EPSG:31028\")\n",
    "spam.crs == agriculture.crs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(agriculture['ID_spam'].isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>ID_lc</th>\n",
       "      <th>geometry</th>\n",
       "      <th>index__spam</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-15.990057</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>1</td>\n",
       "      <td>POINT (394264.774 1845856.132)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-15.989787</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>2</td>\n",
       "      <td>POINT (394293.510 1845855.989)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-15.989518</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>3</td>\n",
       "      <td>POINT (394322.246 1845855.846)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-15.925378</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>4</td>\n",
       "      <td>POINT (401161.317 1845823.013)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-15.925109</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>5</td>\n",
       "      <td>POINT (401190.052 1845822.879)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7654994</th>\n",
       "      <td>-15.940200</td>\n",
       "      <td>15.171060</td>\n",
       "      <td>7654995</td>\n",
       "      <td>POINT (398822.078 1677384.290)</td>\n",
       "      <td>458.0</td>\n",
       "      <td>459.0</td>\n",
       "      <td>670742.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7654995</th>\n",
       "      <td>-15.939930</td>\n",
       "      <td>15.171060</td>\n",
       "      <td>7654996</td>\n",
       "      <td>POINT (398851.085 1677384.166)</td>\n",
       "      <td>458.0</td>\n",
       "      <td>459.0</td>\n",
       "      <td>670742.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7654996</th>\n",
       "      <td>-15.939660</td>\n",
       "      <td>15.171060</td>\n",
       "      <td>7654997</td>\n",
       "      <td>POINT (398880.092 1677384.041)</td>\n",
       "      <td>458.0</td>\n",
       "      <td>459.0</td>\n",
       "      <td>670742.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7654997</th>\n",
       "      <td>-15.939390</td>\n",
       "      <td>15.171060</td>\n",
       "      <td>7654998</td>\n",
       "      <td>POINT (398909.100 1677383.917)</td>\n",
       "      <td>458.0</td>\n",
       "      <td>459.0</td>\n",
       "      <td>670742.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7654998</th>\n",
       "      <td>-15.939120</td>\n",
       "      <td>15.171060</td>\n",
       "      <td>7654999</td>\n",
       "      <td>POINT (398938.107 1677383.792)</td>\n",
       "      <td>458.0</td>\n",
       "      <td>459.0</td>\n",
       "      <td>670742.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7654999 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 x          y    ID_lc                        geometry  \\\n",
       "0       -15.990057  16.693707        1  POINT (394264.774 1845856.132)   \n",
       "1       -15.989787  16.693707        2  POINT (394293.510 1845855.989)   \n",
       "2       -15.989518  16.693707        3  POINT (394322.246 1845855.846)   \n",
       "3       -15.925378  16.693707        4  POINT (401161.317 1845823.013)   \n",
       "4       -15.925109  16.693707        5  POINT (401190.052 1845822.879)   \n",
       "...            ...        ...      ...                             ...   \n",
       "7654994 -15.940200  15.171060  7654995  POINT (398822.078 1677384.290)   \n",
       "7654995 -15.939930  15.171060  7654996  POINT (398851.085 1677384.166)   \n",
       "7654996 -15.939660  15.171060  7654997  POINT (398880.092 1677384.041)   \n",
       "7654997 -15.939390  15.171060  7654998  POINT (398909.100 1677383.917)   \n",
       "7654998 -15.939120  15.171060  7654999  POINT (398938.107 1677383.792)   \n",
       "\n",
       "         index__spam  ID_spam       val  \n",
       "0                NaN      NaN       1.0  \n",
       "1                NaN      NaN       1.0  \n",
       "2                NaN      NaN       1.0  \n",
       "3                NaN      NaN       1.0  \n",
       "4                NaN      NaN       1.0  \n",
       "...              ...      ...       ...  \n",
       "7654994        458.0    459.0  670742.0  \n",
       "7654995        458.0    459.0  670742.0  \n",
       "7654996        458.0    459.0  670742.0  \n",
       "7654997        458.0    459.0  670742.0  \n",
       "7654998        458.0    459.0  670742.0  \n",
       "\n",
       "[7654999 rows x 7 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Not relevant because land cover was already clipped by non-null spam in Arc. \n",
    "# agriculture['val'] = agriculture['val'].fillna(1) # Assign a value of one to non-matches.\n",
    "# agriculture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Land cover that doesn't have a SPAM grid was assigned a value of 1. Now need to provide points for SPAM data where there is no land cover pixel.\n",
    "#### Create xy fields from spam centroids now for later.  Need WGS84 points for all SPAM grids that don't have any land cover points to map to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "spam_xy = spam.to_crs(\"EPSG:31028\") # Geographic CRSs can cause incorrect centroid measures. Use projected.\n",
    "spam_xy['x'] = spam_xy.centroid.x\n",
    "spam_xy['y'] = spam_xy.centroid.y\n",
    "spam_xy.to_csv(os.path.join(out_pth, 'spam_xy.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>geometry</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1387</td>\n",
       "      <td>50698</td>\n",
       "      <td>POINT (720473.131 1525577.307)</td>\n",
       "      <td>720473.130737</td>\n",
       "      <td>1.525577e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2448</td>\n",
       "      <td>320005</td>\n",
       "      <td>POINT (830855.283 1342176.354)</td>\n",
       "      <td>830855.283210</td>\n",
       "      <td>1.342176e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>365</td>\n",
       "      <td>4503</td>\n",
       "      <td>POINT (665153.151 1709568.014)</td>\n",
       "      <td>665153.150572</td>\n",
       "      <td>1.709568e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2150</td>\n",
       "      <td>152650</td>\n",
       "      <td>POINT (730654.504 1387334.758)</td>\n",
       "      <td>730654.503815</td>\n",
       "      <td>1.387335e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1961</td>\n",
       "      <td>105340</td>\n",
       "      <td>POINT (332415.367 1414550.077)</td>\n",
       "      <td>332415.367100</td>\n",
       "      <td>1.414550e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2258</th>\n",
       "      <td>2258</td>\n",
       "      <td>695</td>\n",
       "      <td>468582</td>\n",
       "      <td>POINT (782318.406 1636890.692)</td>\n",
       "      <td>782318.405503</td>\n",
       "      <td>1.636891e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2259</th>\n",
       "      <td>2259</td>\n",
       "      <td>1431</td>\n",
       "      <td>362959</td>\n",
       "      <td>POINT (594366.185 1515594.966)</td>\n",
       "      <td>594366.185263</td>\n",
       "      <td>1.515595e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2260</th>\n",
       "      <td>2260</td>\n",
       "      <td>1319</td>\n",
       "      <td>232790</td>\n",
       "      <td>POINT (603304.222 1534063.640)</td>\n",
       "      <td>603304.222164</td>\n",
       "      <td>1.534064e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2261</th>\n",
       "      <td>2261</td>\n",
       "      <td>905</td>\n",
       "      <td>1133888</td>\n",
       "      <td>POINT (432393.633 1598462.951)</td>\n",
       "      <td>432393.633494</td>\n",
       "      <td>1.598463e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2262</th>\n",
       "      <td>2262</td>\n",
       "      <td>42</td>\n",
       "      <td>3259</td>\n",
       "      <td>POINT (584189.523 1838192.822)</td>\n",
       "      <td>584189.523177</td>\n",
       "      <td>1.838193e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2263 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0  ID_spam  grid_val                        geometry  \\\n",
       "0              0     1387     50698  POINT (720473.131 1525577.307)   \n",
       "1              1     2448    320005  POINT (830855.283 1342176.354)   \n",
       "2              2      365      4503  POINT (665153.151 1709568.014)   \n",
       "3              3     2150    152650  POINT (730654.504 1387334.758)   \n",
       "4              4     1961    105340  POINT (332415.367 1414550.077)   \n",
       "...          ...      ...       ...                             ...   \n",
       "2258        2258      695    468582  POINT (782318.406 1636890.692)   \n",
       "2259        2259     1431    362959  POINT (594366.185 1515594.966)   \n",
       "2260        2260     1319    232790  POINT (603304.222 1534063.640)   \n",
       "2261        2261      905   1133888  POINT (432393.633 1598462.951)   \n",
       "2262        2262       42      3259  POINT (584189.523 1838192.822)   \n",
       "\n",
       "                  x             y  \n",
       "0     720473.130737  1.525577e+06  \n",
       "1     830855.283210  1.342176e+06  \n",
       "2     665153.150572  1.709568e+06  \n",
       "3     730654.503815  1.387335e+06  \n",
       "4     332415.367100  1.414550e+06  \n",
       "...             ...           ...  \n",
       "2258  782318.405503  1.636891e+06  \n",
       "2259  594366.185263  1.515595e+06  \n",
       "2260  603304.222164  1.534064e+06  \n",
       "2261  432393.633494  1.598463e+06  \n",
       "2262  584189.523177  1.838193e+06  \n",
       "\n",
       "[2263 rows x 6 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create new point dataframe from centroids.\n",
    "spam_xy = os.path.join(out_pth, \"spam_xy.csv\")\n",
    "spam_xy = pd.read_csv(spam_xy)\n",
    "geometry = [Point(xy) for xy in zip(spam_xy.x, spam_xy.y)]\n",
    "crs = \"EPSG:31028\"\n",
    "spam_xy = GeoDataFrame(spam_xy, crs=crs, geometry=geometry) \n",
    "spam_xy\n",
    "\n",
    "# 2263 spam cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 2263 entries, 0 to 2262\n",
      "Data columns (total 9 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   Unnamed: 0  2263 non-null   int64   \n",
      " 1   ID_spam     2263 non-null   int64   \n",
      " 2   grid_val    2263 non-null   int64   \n",
      " 3   geometry    2263 non-null   geometry\n",
      " 4   x           2263 non-null   float64 \n",
      " 5   y           2263 non-null   float64 \n",
      " 6   geom_WGS84  2263 non-null   object  \n",
      " 7   X           2263 non-null   object  \n",
      " 8   Y           2263 non-null   object  \n",
      "dtypes: float64(2), geometry(1), int64(3), object(3)\n",
      "memory usage: 159.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# But we need the xy fields to be in WGS84 for snapping to the graph and matching origins table, so we re-project again.\n",
    "spam_xy = spam_xy.to_crs(\"EPSG:4326\")\n",
    "spam_xy[\"geom_WGS84\"] = spam_xy[\"geometry\"].astype('str')\n",
    "\n",
    "# Strip the geometry column of extra characters\n",
    "spam_xy[\"geom_WGS84\"] = spam_xy[\"geom_WGS84\"].str.strip('POINT ') \n",
    "spam_xy[\"geom_WGS84\"] = spam_xy[\"geom_WGS84\"].str.strip('()')\n",
    "\n",
    "# 3. Split the column into two based on the space between the coordinates.\n",
    "XY_spam = spam_xy[\"geom_WGS84\"].str.split(\" \", expand=True)\n",
    "spam_xy[\"X\"] = XY_spam[0]\n",
    "spam_xy[\"Y\"] = XY_spam[1]\n",
    "\n",
    "spam_xy.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 2263 entries, 0 to 2262\n",
      "Data columns (total 9 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   Unnamed: 0  2263 non-null   int64   \n",
      " 1   ID_spam     2263 non-null   int64   \n",
      " 2   grid_val    2263 non-null   int64   \n",
      " 3   geometry    2263 non-null   geometry\n",
      " 4   x           2263 non-null   float64 \n",
      " 5   y           2263 non-null   float64 \n",
      " 6   geom_WGS84  2263 non-null   object  \n",
      " 7   X           2263 non-null   float64 \n",
      " 8   Y           2263 non-null   float64 \n",
      "dtypes: float64(4), geometry(1), int64(3), object(1)\n",
      "memory usage: 159.2+ KB\n"
     ]
    }
   ],
   "source": [
    "spam_xy['X'] = spam_xy['X'].astype(float)\n",
    "spam_xy['Y'] = spam_xy['Y'].astype(float)\n",
    "spam_xy.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>geometry</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>geom_WGS84</th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1387</td>\n",
       "      <td>50698</td>\n",
       "      <td>POINT (-12.95900 13.79197)</td>\n",
       "      <td>720473.130737</td>\n",
       "      <td>1.525577e+06</td>\n",
       "      <td>-12.959001 13.791973</td>\n",
       "      <td>-12.959001</td>\n",
       "      <td>13.791973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2448</td>\n",
       "      <td>320005</td>\n",
       "      <td>POINT (-11.95901 12.12531)</td>\n",
       "      <td>830855.283210</td>\n",
       "      <td>1.342176e+06</td>\n",
       "      <td>-11.959005 12.125313</td>\n",
       "      <td>-11.959005</td>\n",
       "      <td>12.125313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>365</td>\n",
       "      <td>4503</td>\n",
       "      <td>POINT (-13.45900 15.45863)</td>\n",
       "      <td>665153.150572</td>\n",
       "      <td>1.709568e+06</td>\n",
       "      <td>-13.458999 15.458633</td>\n",
       "      <td>-13.458999</td>\n",
       "      <td>15.458633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2150</td>\n",
       "      <td>152650</td>\n",
       "      <td>POINT (-12.87567 12.54198)</td>\n",
       "      <td>730654.503815</td>\n",
       "      <td>1.387335e+06</td>\n",
       "      <td>-12.875668 12.541978</td>\n",
       "      <td>-12.875668</td>\n",
       "      <td>12.541978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1961</td>\n",
       "      <td>105340</td>\n",
       "      <td>POINT (-16.54232 12.79198)</td>\n",
       "      <td>332415.367100</td>\n",
       "      <td>1.414550e+06</td>\n",
       "      <td>-16.542321 12.791977</td>\n",
       "      <td>-16.542321</td>\n",
       "      <td>12.791977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2258</th>\n",
       "      <td>2258</td>\n",
       "      <td>695</td>\n",
       "      <td>468582</td>\n",
       "      <td>POINT (-12.37567 14.79197)</td>\n",
       "      <td>782318.405503</td>\n",
       "      <td>1.636891e+06</td>\n",
       "      <td>-12.37567 14.791969</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>14.791969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2259</th>\n",
       "      <td>2259</td>\n",
       "      <td>1431</td>\n",
       "      <td>362959</td>\n",
       "      <td>POINT (-14.12566 13.70864)</td>\n",
       "      <td>594366.185263</td>\n",
       "      <td>1.515595e+06</td>\n",
       "      <td>-14.125663 13.70864</td>\n",
       "      <td>-14.125663</td>\n",
       "      <td>13.708640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2260</th>\n",
       "      <td>2260</td>\n",
       "      <td>1319</td>\n",
       "      <td>232790</td>\n",
       "      <td>POINT (-14.04233 13.87531)</td>\n",
       "      <td>603304.222164</td>\n",
       "      <td>1.534064e+06</td>\n",
       "      <td>-14.04233 13.875306</td>\n",
       "      <td>-14.042330</td>\n",
       "      <td>13.875306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2261</th>\n",
       "      <td>2261</td>\n",
       "      <td>905</td>\n",
       "      <td>1133888</td>\n",
       "      <td>POINT (-15.62566 14.45864)</td>\n",
       "      <td>432393.633494</td>\n",
       "      <td>1.598463e+06</td>\n",
       "      <td>-15.625658 14.458637</td>\n",
       "      <td>-15.625658</td>\n",
       "      <td>14.458637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2262</th>\n",
       "      <td>2262</td>\n",
       "      <td>42</td>\n",
       "      <td>3259</td>\n",
       "      <td>POINT (-14.20900 16.62529)</td>\n",
       "      <td>584189.523177</td>\n",
       "      <td>1.838193e+06</td>\n",
       "      <td>-14.208996 16.625295</td>\n",
       "      <td>-14.208996</td>\n",
       "      <td>16.625295</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2263 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0  ID_spam  grid_val                    geometry  \\\n",
       "0              0     1387     50698  POINT (-12.95900 13.79197)   \n",
       "1              1     2448    320005  POINT (-11.95901 12.12531)   \n",
       "2              2      365      4503  POINT (-13.45900 15.45863)   \n",
       "3              3     2150    152650  POINT (-12.87567 12.54198)   \n",
       "4              4     1961    105340  POINT (-16.54232 12.79198)   \n",
       "...          ...      ...       ...                         ...   \n",
       "2258        2258      695    468582  POINT (-12.37567 14.79197)   \n",
       "2259        2259     1431    362959  POINT (-14.12566 13.70864)   \n",
       "2260        2260     1319    232790  POINT (-14.04233 13.87531)   \n",
       "2261        2261      905   1133888  POINT (-15.62566 14.45864)   \n",
       "2262        2262       42      3259  POINT (-14.20900 16.62529)   \n",
       "\n",
       "                  x             y            geom_WGS84          X          Y  \n",
       "0     720473.130737  1.525577e+06  -12.959001 13.791973 -12.959001  13.791973  \n",
       "1     830855.283210  1.342176e+06  -11.959005 12.125313 -11.959005  12.125313  \n",
       "2     665153.150572  1.709568e+06  -13.458999 15.458633 -13.458999  15.458633  \n",
       "3     730654.503815  1.387335e+06  -12.875668 12.541978 -12.875668  12.541978  \n",
       "4     332415.367100  1.414550e+06  -16.542321 12.791977 -16.542321  12.791977  \n",
       "...             ...           ...                   ...        ...        ...  \n",
       "2258  782318.405503  1.636891e+06   -12.37567 14.791969 -12.375670  14.791969  \n",
       "2259  594366.185263  1.515595e+06   -14.125663 13.70864 -14.125663  13.708640  \n",
       "2260  603304.222164  1.534064e+06   -14.04233 13.875306 -14.042330  13.875306  \n",
       "2261  432393.633494  1.598463e+06  -15.625658 14.458637 -15.625658  14.458637  \n",
       "2262  584189.523177  1.838193e+06  -14.208996 16.625295 -14.208996  16.625295  \n",
       "\n",
       "[2263 rows x 9 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spam_xy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This is how we still retain SPAM values where there was no land cover cell to represent it. \n",
    "#### There may be a more efficient method here. sjoin does not allow full outer joins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>geometry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2448</td>\n",
       "      <td>320005</td>\n",
       "      <td>MULTIPOLYGON (((835446.983 1337613.654, 826366...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2384</td>\n",
       "      <td>886860</td>\n",
       "      <td>MULTIPOLYGON (((744591.874 1345950.163, 735519...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>2353</td>\n",
       "      <td>1306428</td>\n",
       "      <td>MULTIPOLYGON (((436289.577 1345005.377, 427223...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>2389</td>\n",
       "      <td>1011610</td>\n",
       "      <td>MULTIPOLYGON (((789960.190 1346360.586, 780885...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>2497</td>\n",
       "      <td>2377126</td>\n",
       "      <td>MULTIPOLYGON (((790129.495 1328999.489, 790050...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2180</th>\n",
       "      <td>5</td>\n",
       "      <td>54822</td>\n",
       "      <td>MULTIPOLYGON (((464226.016 1851883.675, 455343...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2211</th>\n",
       "      <td>2383</td>\n",
       "      <td>290657</td>\n",
       "      <td>MULTIPOLYGON (((726431.713 1347928.814, 726377...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2234</th>\n",
       "      <td>2382</td>\n",
       "      <td>743827</td>\n",
       "      <td>MULTIPOLYGON (((726431.713 1347928.814, 717340...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2238</th>\n",
       "      <td>2345</td>\n",
       "      <td>94605</td>\n",
       "      <td>MULTIPOLYGON (((354684.661 1345293.292, 345616...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2241</th>\n",
       "      <td>2443</td>\n",
       "      <td>1451706</td>\n",
       "      <td>MULTIPOLYGON (((790050.413 1337136.482, 780972...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>123 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      ID_spam  grid_val                                           geometry\n",
       "1        2448    320005  MULTIPOLYGON (((835446.983 1337613.654, 826366...\n",
       "25       2384    886860  MULTIPOLYGON (((744591.874 1345950.163, 735519...\n",
       "54       2353   1306428  MULTIPOLYGON (((436289.577 1345005.377, 427223...\n",
       "62       2389   1011610  MULTIPOLYGON (((789960.190 1346360.586, 780885...\n",
       "90       2497   2377126  MULTIPOLYGON (((790129.495 1328999.489, 790050...\n",
       "...       ...       ...                                                ...\n",
       "2180        5     54822  MULTIPOLYGON (((464226.016 1851883.675, 455343...\n",
       "2211     2383    290657  MULTIPOLYGON (((726431.713 1347928.814, 726377...\n",
       "2234     2382    743827  MULTIPOLYGON (((726431.713 1347928.814, 717340...\n",
       "2238     2345     94605  MULTIPOLYGON (((354684.661 1345293.292, 345616...\n",
       "2241     2443   1451706  MULTIPOLYGON (((790050.413 1337136.482, 780972...\n",
       "\n",
       "[123 rows x 3 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find which SPAM cells were not joined to land cover, create a point for them, and append them to the origins (agriculture) dataset.\n",
    "spam_nomatch = spam[~spam.ID_spam.isin(agriculture.ID_spam)]\n",
    "spam_nomatch\n",
    "\n",
    "# 123 cells without corresponding land cover points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val_x</th>\n",
       "      <th>geometry_x</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>grid_val_y</th>\n",
       "      <th>geometry_y</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>geom_WGS84</th>\n",
       "      <th>X</th>\n",
       "      <th>Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2448</td>\n",
       "      <td>320005</td>\n",
       "      <td>MULTIPOLYGON (((835446.983 1337613.654, 826366...</td>\n",
       "      <td>1</td>\n",
       "      <td>320005</td>\n",
       "      <td>POINT (-11.95901 12.12531)</td>\n",
       "      <td>830855.283210</td>\n",
       "      <td>1.342176e+06</td>\n",
       "      <td>-11.959005 12.125313</td>\n",
       "      <td>-11.959005</td>\n",
       "      <td>12.125313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2384</td>\n",
       "      <td>886860</td>\n",
       "      <td>MULTIPOLYGON (((744591.874 1345950.163, 735519...</td>\n",
       "      <td>25</td>\n",
       "      <td>886860</td>\n",
       "      <td>POINT (-12.79234 12.20865)</td>\n",
       "      <td>740018.245292</td>\n",
       "      <td>1.350524e+06</td>\n",
       "      <td>-12.792335 12.208646</td>\n",
       "      <td>-12.792335</td>\n",
       "      <td>12.208646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2353</td>\n",
       "      <td>1306428</td>\n",
       "      <td>MULTIPOLYGON (((436289.577 1345005.377, 427223...</td>\n",
       "      <td>54</td>\n",
       "      <td>1306428</td>\n",
       "      <td>POINT (-15.62566 12.20865)</td>\n",
       "      <td>431766.963707</td>\n",
       "      <td>1.349623e+06</td>\n",
       "      <td>-15.625658 12.208646</td>\n",
       "      <td>-15.625658</td>\n",
       "      <td>12.208646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2389</td>\n",
       "      <td>1011610</td>\n",
       "      <td>MULTIPOLYGON (((789960.190 1346360.586, 780885...</td>\n",
       "      <td>62</td>\n",
       "      <td>1011610</td>\n",
       "      <td>POINT (-12.37567 12.20865)</td>\n",
       "      <td>785378.175609</td>\n",
       "      <td>1.350928e+06</td>\n",
       "      <td>-12.37567 12.208646</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.208646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2497</td>\n",
       "      <td>2377126</td>\n",
       "      <td>MULTIPOLYGON (((790129.495 1328999.489, 790050...</td>\n",
       "      <td>90</td>\n",
       "      <td>2377126</td>\n",
       "      <td>POINT (-12.29351 12.04963)</td>\n",
       "      <td>794499.340290</td>\n",
       "      <td>1.333415e+06</td>\n",
       "      <td>-12.293506 12.049633</td>\n",
       "      <td>-12.293506</td>\n",
       "      <td>12.049633</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>5</td>\n",
       "      <td>54822</td>\n",
       "      <td>MULTIPOLYGON (((464226.016 1851883.675, 455343...</td>\n",
       "      <td>2180</td>\n",
       "      <td>54822</td>\n",
       "      <td>POINT (-15.37566 16.79196)</td>\n",
       "      <td>459793.687462</td>\n",
       "      <td>1.856501e+06</td>\n",
       "      <td>-15.375659 16.791961</td>\n",
       "      <td>-15.375659</td>\n",
       "      <td>16.791961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>2383</td>\n",
       "      <td>290657</td>\n",
       "      <td>MULTIPOLYGON (((726431.713 1347928.814, 726377...</td>\n",
       "      <td>2211</td>\n",
       "      <td>290657</td>\n",
       "      <td>POINT (-12.87370 12.21204)</td>\n",
       "      <td>731159.466492</td>\n",
       "      <td>1.350829e+06</td>\n",
       "      <td>-12.873696 12.212042</td>\n",
       "      <td>-12.873696</td>\n",
       "      <td>12.212042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>2382</td>\n",
       "      <td>743827</td>\n",
       "      <td>MULTIPOLYGON (((726431.713 1347928.814, 717340...</td>\n",
       "      <td>2234</td>\n",
       "      <td>743827</td>\n",
       "      <td>POINT (-12.95561 12.22402)</td>\n",
       "      <td>722233.491819</td>\n",
       "      <td>1.352085e+06</td>\n",
       "      <td>-12.955614 12.224018</td>\n",
       "      <td>-12.955614</td>\n",
       "      <td>12.224018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>2345</td>\n",
       "      <td>94605</td>\n",
       "      <td>MULTIPOLYGON (((354684.661 1345293.292, 345616...</td>\n",
       "      <td>2238</td>\n",
       "      <td>94605</td>\n",
       "      <td>POINT (-16.37565 12.20865)</td>\n",
       "      <td>350173.846532</td>\n",
       "      <td>1.349925e+06</td>\n",
       "      <td>-16.375655 12.208646</td>\n",
       "      <td>-16.375655</td>\n",
       "      <td>12.208646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>2443</td>\n",
       "      <td>1451706</td>\n",
       "      <td>MULTIPOLYGON (((790050.413 1337136.482, 780972...</td>\n",
       "      <td>2241</td>\n",
       "      <td>1451706</td>\n",
       "      <td>POINT (-12.37567 12.12531)</td>\n",
       "      <td>785467.284593</td>\n",
       "      <td>1.341705e+06</td>\n",
       "      <td>-12.37567 12.125313</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.125313</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>123 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     ID_spam  grid_val_x                                         geometry_x  \\\n",
       "0       2448      320005  MULTIPOLYGON (((835446.983 1337613.654, 826366...   \n",
       "1       2384      886860  MULTIPOLYGON (((744591.874 1345950.163, 735519...   \n",
       "2       2353     1306428  MULTIPOLYGON (((436289.577 1345005.377, 427223...   \n",
       "3       2389     1011610  MULTIPOLYGON (((789960.190 1346360.586, 780885...   \n",
       "4       2497     2377126  MULTIPOLYGON (((790129.495 1328999.489, 790050...   \n",
       "..       ...         ...                                                ...   \n",
       "118        5       54822  MULTIPOLYGON (((464226.016 1851883.675, 455343...   \n",
       "119     2383      290657  MULTIPOLYGON (((726431.713 1347928.814, 726377...   \n",
       "120     2382      743827  MULTIPOLYGON (((726431.713 1347928.814, 717340...   \n",
       "121     2345       94605  MULTIPOLYGON (((354684.661 1345293.292, 345616...   \n",
       "122     2443     1451706  MULTIPOLYGON (((790050.413 1337136.482, 780972...   \n",
       "\n",
       "     Unnamed: 0  grid_val_y                  geometry_y              x  \\\n",
       "0             1      320005  POINT (-11.95901 12.12531)  830855.283210   \n",
       "1            25      886860  POINT (-12.79234 12.20865)  740018.245292   \n",
       "2            54     1306428  POINT (-15.62566 12.20865)  431766.963707   \n",
       "3            62     1011610  POINT (-12.37567 12.20865)  785378.175609   \n",
       "4            90     2377126  POINT (-12.29351 12.04963)  794499.340290   \n",
       "..          ...         ...                         ...            ...   \n",
       "118        2180       54822  POINT (-15.37566 16.79196)  459793.687462   \n",
       "119        2211      290657  POINT (-12.87370 12.21204)  731159.466492   \n",
       "120        2234      743827  POINT (-12.95561 12.22402)  722233.491819   \n",
       "121        2238       94605  POINT (-16.37565 12.20865)  350173.846532   \n",
       "122        2241     1451706  POINT (-12.37567 12.12531)  785467.284593   \n",
       "\n",
       "                y            geom_WGS84          X          Y  \n",
       "0    1.342176e+06  -11.959005 12.125313 -11.959005  12.125313  \n",
       "1    1.350524e+06  -12.792335 12.208646 -12.792335  12.208646  \n",
       "2    1.349623e+06  -15.625658 12.208646 -15.625658  12.208646  \n",
       "3    1.350928e+06   -12.37567 12.208646 -12.375670  12.208646  \n",
       "4    1.333415e+06  -12.293506 12.049633 -12.293506  12.049633  \n",
       "..            ...                   ...        ...        ...  \n",
       "118  1.856501e+06  -15.375659 16.791961 -15.375659  16.791961  \n",
       "119  1.350829e+06  -12.873696 12.212042 -12.873696  12.212042  \n",
       "120  1.352085e+06  -12.955614 12.224018 -12.955614  12.224018  \n",
       "121  1.349925e+06  -16.375655 12.208646 -16.375655  12.208646  \n",
       "122  1.341705e+06   -12.37567 12.125313 -12.375670  12.125313  \n",
       "\n",
       "[123 rows x 11 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spam_nomatch = pd.merge(spam_nomatch, spam_xy, on='ID_spam',how='left') # Add the WGS84 xy fields created earlier only for SPAM grids that didn't have land cover points.\n",
    "spam_nomatch\n",
    "# Resulting dataset should have same number of rows as earlier spam_nomatch table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 123 entries, 0 to 122\n",
      "Data columns (total 11 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   ID_spam     123 non-null    int64   \n",
      " 1   grid_val_x  123 non-null    int64   \n",
      " 2   geometry_x  123 non-null    geometry\n",
      " 3   Unnamed: 0  123 non-null    int64   \n",
      " 4   grid_val_y  123 non-null    int64   \n",
      " 5   geometry_y  123 non-null    geometry\n",
      " 6   x           123 non-null    float64 \n",
      " 7   y           123 non-null    float64 \n",
      " 8   geom_WGS84  123 non-null    object  \n",
      " 9   X           123 non-null    float64 \n",
      " 10  Y           123 non-null    float64 \n",
      "dtypes: float64(4), geometry(2), int64(4), object(1)\n",
      "memory usage: 11.5+ KB\n"
     ]
    }
   ],
   "source": [
    "spam_nomatch.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 123 entries, 0 to 122\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   ID_spam     123 non-null    int64   \n",
      " 1   grid_val    123 non-null    int64   \n",
      " 2   geom_UTM    123 non-null    geometry\n",
      " 3   geom_WGS84  123 non-null    geometry\n",
      " 4   x           123 non-null    float64 \n",
      " 5   y           123 non-null    float64 \n",
      "dtypes: float64(2), geometry(2), int64(2)\n",
      "memory usage: 6.7 KB\n"
     ]
    }
   ],
   "source": [
    "# Clean up the no match dataset prior to appending to final origins file. Make sure XY fields are exact matches, including case.\n",
    "spam_nomatch = spam_nomatch.drop(columns=['Unnamed: 0', 'grid_val_y', 'geom_WGS84', 'x', 'y'])\n",
    "spam_nomatch.rename(columns={'grid_val_x':'grid_val', 'geometry_x':'geom_UTM', 'geometry_y':'geom_WGS84', 'X':'x', 'Y':'y'}, inplace=True)\n",
    "spam_nomatch.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 123 entries, 0 to 122\n",
      "Data columns (total 7 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  123 non-null    int64  \n",
      " 1   ID_spam     123 non-null    int64  \n",
      " 2   grid_val    123 non-null    int64  \n",
      " 3   geom_UTM    123 non-null    object \n",
      " 4   geom_WGS84  123 non-null    object \n",
      " 5   x           123 non-null    float64\n",
      " 6   y           123 non-null    float64\n",
      "dtypes: float64(2), int64(3), object(2)\n",
      "memory usage: 6.9+ KB\n"
     ]
    }
   ],
   "source": [
    "# Just in case, I want to reload the no matches as a regular dataframe and make sure all data types are appropriate.\n",
    "spam_nomatch.to_csv(os.path.join(out_pth, 'spam_nomatch.csv'))\n",
    "spam_nomatch = os.path.join(out_pth, \"spam_nomatch.csv\")\n",
    "spam_nomatch = pd.read_csv(spam_nomatch)\n",
    "spam_nomatch.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "agriculture = agriculture.to_crs(\"EPSG:4326\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13941001 entries, 0 to 13941000\n",
      "Data columns (total 8 columns):\n",
      " #   Column      Dtype   \n",
      "---  ------      -----   \n",
      " 0   ID_spam     float64 \n",
      " 1   grid_val    float64 \n",
      " 2   ID_LC       float64 \n",
      " 3   val         float64 \n",
      " 4   geometry    geometry\n",
      " 5   geom_WGS84  object  \n",
      " 6   x           float64 \n",
      " 7   y           float64 \n",
      "dtypes: float64(6), geometry(1), object(1)\n",
      "memory usage: 850.9+ MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>geometry</th>\n",
       "      <th>geom_WGS84</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.56250 16.67094)</td>\n",
       "      <td>-15.562504 16.670935</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.55442 16.68225)</td>\n",
       "      <td>-15.554419 16.682254</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.55981 16.67686)</td>\n",
       "      <td>-15.559809 16.676864</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.56196 16.67094)</td>\n",
       "      <td>-15.561965 16.670935</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.57113 16.66824)</td>\n",
       "      <td>-15.571128 16.66824</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13940996</th>\n",
       "      <td>2342.0</td>\n",
       "      <td>455853.0</td>\n",
       "      <td>249666.0</td>\n",
       "      <td>45585.300000</td>\n",
       "      <td>POINT (-11.49260 12.33046)</td>\n",
       "      <td>-11.492597 12.330455</td>\n",
       "      <td>-11.492597</td>\n",
       "      <td>12.330455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13940997</th>\n",
       "      <td>2342.0</td>\n",
       "      <td>455853.0</td>\n",
       "      <td>249665.0</td>\n",
       "      <td>45585.300000</td>\n",
       "      <td>POINT (-11.49314 12.33046)</td>\n",
       "      <td>-11.493136 12.330455</td>\n",
       "      <td>-11.493136</td>\n",
       "      <td>12.330455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13940998</th>\n",
       "      <td>2342.0</td>\n",
       "      <td>455853.0</td>\n",
       "      <td>249664.0</td>\n",
       "      <td>45585.300000</td>\n",
       "      <td>POINT (-11.48882 12.33207)</td>\n",
       "      <td>-11.488824 12.332072</td>\n",
       "      <td>-11.488824</td>\n",
       "      <td>12.332072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13940999</th>\n",
       "      <td>2342.0</td>\n",
       "      <td>455853.0</td>\n",
       "      <td>249661.0</td>\n",
       "      <td>45585.300000</td>\n",
       "      <td>POINT (-11.48936 12.33261)</td>\n",
       "      <td>-11.489363 12.332611</td>\n",
       "      <td>-11.489363</td>\n",
       "      <td>12.332611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941000</th>\n",
       "      <td>2342.0</td>\n",
       "      <td>455853.0</td>\n",
       "      <td>249663.0</td>\n",
       "      <td>45585.300000</td>\n",
       "      <td>POINT (-11.48936 12.33207)</td>\n",
       "      <td>-11.489363 12.332072</td>\n",
       "      <td>-11.489363</td>\n",
       "      <td>12.332072</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13941001 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          ID_spam  grid_val       ID_LC           val  \\\n",
       "0             9.0    2487.0  13787187.0      4.236797   \n",
       "1             9.0    2487.0  13818608.0      4.236797   \n",
       "2             9.0    2487.0  13787063.0      4.236797   \n",
       "3             9.0    2487.0  13787188.0      4.236797   \n",
       "4             9.0    2487.0  13787225.0      4.236797   \n",
       "...           ...       ...         ...           ...   \n",
       "13940996   2342.0  455853.0    249666.0  45585.300000   \n",
       "13940997   2342.0  455853.0    249665.0  45585.300000   \n",
       "13940998   2342.0  455853.0    249664.0  45585.300000   \n",
       "13940999   2342.0  455853.0    249661.0  45585.300000   \n",
       "13941000   2342.0  455853.0    249663.0  45585.300000   \n",
       "\n",
       "                            geometry            geom_WGS84          x  \\\n",
       "0         POINT (-15.56250 16.67094)  -15.562504 16.670935 -15.562504   \n",
       "1         POINT (-15.55442 16.68225)  -15.554419 16.682254 -15.554419   \n",
       "2         POINT (-15.55981 16.67686)  -15.559809 16.676864 -15.559809   \n",
       "3         POINT (-15.56196 16.67094)  -15.561965 16.670935 -15.561965   \n",
       "4         POINT (-15.57113 16.66824)   -15.571128 16.66824 -15.571128   \n",
       "...                              ...                   ...        ...   \n",
       "13940996  POINT (-11.49260 12.33046)  -11.492597 12.330455 -11.492597   \n",
       "13940997  POINT (-11.49314 12.33046)  -11.493136 12.330455 -11.493136   \n",
       "13940998  POINT (-11.48882 12.33207)  -11.488824 12.332072 -11.488824   \n",
       "13940999  POINT (-11.48936 12.33261)  -11.489363 12.332611 -11.489363   \n",
       "13941000  POINT (-11.48936 12.33207)  -11.489363 12.332072 -11.489363   \n",
       "\n",
       "                  y  \n",
       "0         16.670935  \n",
       "1         16.682254  \n",
       "2         16.676864  \n",
       "3         16.670935  \n",
       "4         16.668240  \n",
       "...             ...  \n",
       "13940996  12.330455  \n",
       "13940997  12.330455  \n",
       "13940998  12.332072  \n",
       "13940999  12.332611  \n",
       "13941000  12.332072  \n",
       "\n",
       "[13941001 rows x 8 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture[\"geom_WGS84\"] = agriculture[\"geometry\"].astype('str')\n",
    "\n",
    "# Strip the geometry column of extra characters\n",
    "agriculture[\"geom_WGS84\"] = agriculture[\"geom_WGS84\"].str.strip('POINT ') \n",
    "agriculture[\"geom_WGS84\"] = agriculture[\"geom_WGS84\"].str.strip('()')\n",
    "\n",
    "# 3. Split the column into two based on the space between the coordinates.\n",
    "XY_ag = agriculture[\"geom_WGS84\"].str.split(\" \", expand=True)\n",
    "agriculture[\"x\"] = XY_ag[0]\n",
    "agriculture[\"y\"] = XY_ag[1]\n",
    "\n",
    "agriculture['x'] = agriculture['x'].astype(float)\n",
    "agriculture['y'] = agriculture['y'].astype(float)\n",
    "agriculture.info()\n",
    "agriculture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(agriculture['x'].isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>geometry</th>\n",
       "      <th>geom_WGS84</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>geom_UTM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.56250 16.67094)</td>\n",
       "      <td>-15.562504 16.670935</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.55442 16.68225)</td>\n",
       "      <td>-15.554419 16.682254</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.55981 16.67686)</td>\n",
       "      <td>-15.559809 16.676864</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.56196 16.67094)</td>\n",
       "      <td>-15.561965 16.670935</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>POINT (-15.57113 16.66824)</td>\n",
       "      <td>-15.571128 16.66824</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>5.0</td>\n",
       "      <td>54822.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>POINT (-15.375658562426967 16.791960774825146)</td>\n",
       "      <td>-15.375659</td>\n",
       "      <td>16.791961</td>\n",
       "      <td>118.0</td>\n",
       "      <td>MULTIPOLYGON (((464226.0159186065 1851883.6752...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>2383.0</td>\n",
       "      <td>290657.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>POINT (-12.873695714743537 12.212042461981278)</td>\n",
       "      <td>-12.873696</td>\n",
       "      <td>12.212042</td>\n",
       "      <td>119.0</td>\n",
       "      <td>MULTIPOLYGON (((726431.7126558168 1347928.8139...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>2382.0</td>\n",
       "      <td>743827.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>POINT (-12.955613789985883 12.224017589494675)</td>\n",
       "      <td>-12.955614</td>\n",
       "      <td>12.224018</td>\n",
       "      <td>120.0</td>\n",
       "      <td>MULTIPOLYGON (((726431.7126558168 1347928.8139...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>2345.0</td>\n",
       "      <td>94605.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>POINT (-16.37565473031309 12.2086455435392)</td>\n",
       "      <td>-16.375655</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>121.0</td>\n",
       "      <td>MULTIPOLYGON (((354684.6609952422 1345293.2921...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>2443.0</td>\n",
       "      <td>1451706.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>POINT (-12.375670088932086 12.125312535564268)</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>122.0</td>\n",
       "      <td>MULTIPOLYGON (((790050.4125279284 1337136.4817...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13941124 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     ID_spam   grid_val       ID_LC       val                    geometry  \\\n",
       "0        9.0     2487.0  13787187.0  4.236797  POINT (-15.56250 16.67094)   \n",
       "1        9.0     2487.0  13818608.0  4.236797  POINT (-15.55442 16.68225)   \n",
       "2        9.0     2487.0  13787063.0  4.236797  POINT (-15.55981 16.67686)   \n",
       "3        9.0     2487.0  13787188.0  4.236797  POINT (-15.56196 16.67094)   \n",
       "4        9.0     2487.0  13787225.0  4.236797  POINT (-15.57113 16.66824)   \n",
       "..       ...        ...         ...       ...                         ...   \n",
       "118      5.0    54822.0         NaN       NaN                        None   \n",
       "119   2383.0   290657.0         NaN       NaN                        None   \n",
       "120   2382.0   743827.0         NaN       NaN                        None   \n",
       "121   2345.0    94605.0         NaN       NaN                        None   \n",
       "122   2443.0  1451706.0         NaN       NaN                        None   \n",
       "\n",
       "                                         geom_WGS84          x          y  \\\n",
       "0                              -15.562504 16.670935 -15.562504  16.670935   \n",
       "1                              -15.554419 16.682254 -15.554419  16.682254   \n",
       "2                              -15.559809 16.676864 -15.559809  16.676864   \n",
       "3                              -15.561965 16.670935 -15.561965  16.670935   \n",
       "4                               -15.571128 16.66824 -15.571128  16.668240   \n",
       "..                                              ...        ...        ...   \n",
       "118  POINT (-15.375658562426967 16.791960774825146) -15.375659  16.791961   \n",
       "119  POINT (-12.873695714743537 12.212042461981278) -12.873696  12.212042   \n",
       "120  POINT (-12.955613789985883 12.224017589494675) -12.955614  12.224018   \n",
       "121     POINT (-16.37565473031309 12.2086455435392) -16.375655  12.208646   \n",
       "122  POINT (-12.375670088932086 12.125312535564268) -12.375670  12.125313   \n",
       "\n",
       "     Unnamed: 0                                           geom_UTM  \n",
       "0           NaN                                                NaN  \n",
       "1           NaN                                                NaN  \n",
       "2           NaN                                                NaN  \n",
       "3           NaN                                                NaN  \n",
       "4           NaN                                                NaN  \n",
       "..          ...                                                ...  \n",
       "118       118.0  MULTIPOLYGON (((464226.0159186065 1851883.6752...  \n",
       "119       119.0  MULTIPOLYGON (((726431.7126558168 1347928.8139...  \n",
       "120       120.0  MULTIPOLYGON (((726431.7126558168 1347928.8139...  \n",
       "121       121.0  MULTIPOLYGON (((354684.6609952422 1345293.2921...  \n",
       "122       122.0  MULTIPOLYGON (((790050.4125279284 1337136.4817...  \n",
       "\n",
       "[13941124 rows x 10 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture = agriculture.append(spam_nomatch) \n",
    "agriculture\n",
    "# Make sure XY fields appended onto the agriculture XY fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 13941124 entries, 0 to 122\n",
      "Data columns (total 10 columns):\n",
      " #   Column      Dtype   \n",
      "---  ------      -----   \n",
      " 0   ID_spam     float64 \n",
      " 1   grid_val    float64 \n",
      " 2   ID_LC       float64 \n",
      " 3   val         float64 \n",
      " 4   geometry    geometry\n",
      " 5   geom_WGS84  object  \n",
      " 6   x           float64 \n",
      " 7   y           float64 \n",
      " 8   Unnamed: 0  float64 \n",
      " 9   geom_UTM    object  \n",
      "dtypes: float64(7), geometry(1), object(2)\n",
      "memory usage: 1.1+ GB\n"
     ]
    }
   ],
   "source": [
    "agriculture.info()\n",
    "# 13941124 rows in result, - 123 nomatches = number of rows in original agriculture dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "agriculture = agriculture.drop(columns=['geometry','geom_UTM', 'geom_WGS84'])\n",
    "agriculture['x'] = agriculture['x'].astype(float) # Just in case.\n",
    "agriculture['y'] = agriculture['y'].astype(float)\n",
    "agriculture['ID_LC'] = agriculture['ID_LC'].astype(float)\n",
    "agriculture['ID_spam'] = agriculture['ID_spam'].astype(float)\n",
    "agriculture['val'] = agriculture['val'].astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Give a dummy ID for SPAM locations that don't have land cover pixels, and vice versa.\n",
    "agriculture['ID_LC'] = agriculture['ID_LC'].fillna(0.1)\n",
    "agriculture['ID_spam'] = agriculture['ID_spam'].fillna(0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787138.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.673091</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787238.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.581368</td>\n",
       "      <td>16.667162</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818609.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.553880</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818411.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.544717</td>\n",
       "      <td>16.691956</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787224.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571667</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818566.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558192</td>\n",
       "      <td>16.683871</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818621.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.553880</td>\n",
       "      <td>16.681715</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787064.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559270</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818414.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.551185</td>\n",
       "      <td>16.691417</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787065.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.567894</td>\n",
       "      <td>16.676325</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818605.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.556036</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818520.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.525314</td>\n",
       "      <td>16.686566</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818604.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.556575</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818380.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.545795</td>\n",
       "      <td>16.693034</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818579.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.520463</td>\n",
       "      <td>16.683871</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818379.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.546334</td>\n",
       "      <td>16.693034</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787036.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559270</td>\n",
       "      <td>16.679559</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818446.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.549568</td>\n",
       "      <td>16.690339</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818445.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.550107</td>\n",
       "      <td>16.690339</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818523.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.521002</td>\n",
       "      <td>16.686566</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818421.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.547412</td>\n",
       "      <td>16.691417</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787237.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.564121</td>\n",
       "      <td>16.667701</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818606.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.555497</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818580.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558192</td>\n",
       "      <td>16.683332</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787180.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561426</td>\n",
       "      <td>16.671474</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787020.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.570050</td>\n",
       "      <td>16.684410</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818694.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.509683</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787035.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558731</td>\n",
       "      <td>16.680098</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787184.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.567355</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787189.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561426</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787058.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558731</td>\n",
       "      <td>16.677403</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787223.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.572206</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787236.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.564660</td>\n",
       "      <td>16.667701</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818378.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.546873</td>\n",
       "      <td>16.693034</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818420.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.547951</td>\n",
       "      <td>16.691417</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ID_spam  grid_val       ID_LC       val          x          y  Unnamed: 0\n",
       "0       9.0    2487.0  13787187.0  4.236797 -15.562504  16.670935         NaN\n",
       "1       9.0    2487.0  13818608.0  4.236797 -15.554419  16.682254         NaN\n",
       "2       9.0    2487.0  13787063.0  4.236797 -15.559809  16.676864         NaN\n",
       "3       9.0    2487.0  13787188.0  4.236797 -15.561965  16.670935         NaN\n",
       "4       9.0    2487.0  13787225.0  4.236797 -15.571128  16.668240         NaN\n",
       "5       9.0    2487.0  13787138.0  4.236797 -15.561965  16.673091         NaN\n",
       "6       9.0    2487.0  13787238.0  4.236797 -15.581368  16.667162         NaN\n",
       "7       9.0    2487.0  13818609.0  4.236797 -15.553880  16.682254         NaN\n",
       "8       9.0    2487.0  13818411.0  4.236797 -15.544717  16.691956         NaN\n",
       "9       9.0    2487.0  13787224.0  4.236797 -15.571667  16.668240         NaN\n",
       "10      9.0    2487.0  13818566.0  4.236797 -15.558192  16.683871         NaN\n",
       "11      9.0    2487.0  13818621.0  4.236797 -15.553880  16.681715         NaN\n",
       "12      9.0    2487.0  13787064.0  4.236797 -15.559270  16.676864         NaN\n",
       "13      9.0    2487.0  13818414.0  4.236797 -15.551185  16.691417         NaN\n",
       "14      9.0    2487.0  13787065.0  4.236797 -15.567894  16.676325         NaN\n",
       "15      9.0    2487.0  13818605.0  4.236797 -15.556036  16.682254         NaN\n",
       "16      9.0    2487.0  13818520.0  4.236797 -15.525314  16.686566         NaN\n",
       "17      9.0    2487.0  13818604.0  4.236797 -15.556575  16.682254         NaN\n",
       "18      9.0    2487.0  13818380.0  4.236797 -15.545795  16.693034         NaN\n",
       "19      9.0    2487.0  13818579.0  4.236797 -15.520463  16.683871         NaN\n",
       "20      9.0    2487.0  13818379.0  4.236797 -15.546334  16.693034         NaN\n",
       "21      9.0    2487.0  13787036.0  4.236797 -15.559270  16.679559         NaN\n",
       "22      9.0    2487.0  13818446.0  4.236797 -15.549568  16.690339         NaN\n",
       "23      9.0    2487.0  13818445.0  4.236797 -15.550107  16.690339         NaN\n",
       "24      9.0    2487.0  13818523.0  4.236797 -15.521002  16.686566         NaN\n",
       "25      9.0    2487.0  13818421.0  4.236797 -15.547412  16.691417         NaN\n",
       "26      9.0    2487.0  13787237.0  4.236797 -15.564121  16.667701         NaN\n",
       "27      9.0    2487.0  13818606.0  4.236797 -15.555497  16.682254         NaN\n",
       "28      9.0    2487.0  13818580.0  4.236797 -15.558192  16.683332         NaN\n",
       "29      9.0    2487.0  13787180.0  4.236797 -15.561426  16.671474         NaN\n",
       "30      9.0    2487.0  13787020.0  4.236797 -15.570050  16.684410         NaN\n",
       "31      9.0    2487.0  13818694.0  4.236797 -15.509683  16.670935         NaN\n",
       "32      9.0    2487.0  13787035.0  4.236797 -15.558731  16.680098         NaN\n",
       "33      9.0    2487.0  13787184.0  4.236797 -15.567355  16.670935         NaN\n",
       "34      9.0    2487.0  13787189.0  4.236797 -15.561426  16.670935         NaN\n",
       "35      9.0    2487.0  13787058.0  4.236797 -15.558731  16.677403         NaN\n",
       "36      9.0    2487.0  13787223.0  4.236797 -15.572206  16.668240         NaN\n",
       "37      9.0    2487.0  13787236.0  4.236797 -15.564660  16.667701         NaN\n",
       "38      9.0    2487.0  13818378.0  4.236797 -15.546873  16.693034         NaN\n",
       "39      9.0    2487.0  13818420.0  4.236797 -15.547951  16.691417         NaN"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture.head(40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787138.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.673091</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787238.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.581368</td>\n",
       "      <td>16.667162</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818609.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.553880</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818411.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.544717</td>\n",
       "      <td>16.691956</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787224.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571667</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818566.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558192</td>\n",
       "      <td>16.683871</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818621.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.553880</td>\n",
       "      <td>16.681715</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787064.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559270</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818414.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.551185</td>\n",
       "      <td>16.691417</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787065.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.567894</td>\n",
       "      <td>16.676325</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818605.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.556036</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818520.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.525314</td>\n",
       "      <td>16.686566</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818604.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.556575</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818380.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.545795</td>\n",
       "      <td>16.693034</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818579.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.520463</td>\n",
       "      <td>16.683871</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818379.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.546334</td>\n",
       "      <td>16.693034</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787036.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559270</td>\n",
       "      <td>16.679559</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818446.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.549568</td>\n",
       "      <td>16.690339</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818445.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.550107</td>\n",
       "      <td>16.690339</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818523.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.521002</td>\n",
       "      <td>16.686566</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818421.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.547412</td>\n",
       "      <td>16.691417</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787237.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.564121</td>\n",
       "      <td>16.667701</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818606.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.555497</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818580.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558192</td>\n",
       "      <td>16.683332</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787180.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561426</td>\n",
       "      <td>16.671474</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ID_spam  grid_val       ID_LC       val          x          y  Unnamed: 0\n",
       "0       9.0    2487.0  13787187.0  4.236797 -15.562504  16.670935         NaN\n",
       "1       9.0    2487.0  13818608.0  4.236797 -15.554419  16.682254         NaN\n",
       "2       9.0    2487.0  13787063.0  4.236797 -15.559809  16.676864         NaN\n",
       "3       9.0    2487.0  13787188.0  4.236797 -15.561965  16.670935         NaN\n",
       "4       9.0    2487.0  13787225.0  4.236797 -15.571128  16.668240         NaN\n",
       "5       9.0    2487.0  13787138.0  4.236797 -15.561965  16.673091         NaN\n",
       "6       9.0    2487.0  13787238.0  4.236797 -15.581368  16.667162         NaN\n",
       "7       9.0    2487.0  13818609.0  4.236797 -15.553880  16.682254         NaN\n",
       "8       9.0    2487.0  13818411.0  4.236797 -15.544717  16.691956         NaN\n",
       "9       9.0    2487.0  13787224.0  4.236797 -15.571667  16.668240         NaN\n",
       "10      9.0    2487.0  13818566.0  4.236797 -15.558192  16.683871         NaN\n",
       "11      9.0    2487.0  13818621.0  4.236797 -15.553880  16.681715         NaN\n",
       "12      9.0    2487.0  13787064.0  4.236797 -15.559270  16.676864         NaN\n",
       "13      9.0    2487.0  13818414.0  4.236797 -15.551185  16.691417         NaN\n",
       "14      9.0    2487.0  13787065.0  4.236797 -15.567894  16.676325         NaN\n",
       "15      9.0    2487.0  13818605.0  4.236797 -15.556036  16.682254         NaN\n",
       "16      9.0    2487.0  13818520.0  4.236797 -15.525314  16.686566         NaN\n",
       "17      9.0    2487.0  13818604.0  4.236797 -15.556575  16.682254         NaN\n",
       "18      9.0    2487.0  13818380.0  4.236797 -15.545795  16.693034         NaN\n",
       "19      9.0    2487.0  13818579.0  4.236797 -15.520463  16.683871         NaN\n",
       "20      9.0    2487.0  13818379.0  4.236797 -15.546334  16.693034         NaN\n",
       "21      9.0    2487.0  13787036.0  4.236797 -15.559270  16.679559         NaN\n",
       "22      9.0    2487.0  13818446.0  4.236797 -15.549568  16.690339         NaN\n",
       "23      9.0    2487.0  13818445.0  4.236797 -15.550107  16.690339         NaN\n",
       "24      9.0    2487.0  13818523.0  4.236797 -15.521002  16.686566         NaN\n",
       "25      9.0    2487.0  13818421.0  4.236797 -15.547412  16.691417         NaN\n",
       "26      9.0    2487.0  13787237.0  4.236797 -15.564121  16.667701         NaN\n",
       "27      9.0    2487.0  13818606.0  4.236797 -15.555497  16.682254         NaN\n",
       "28      9.0    2487.0  13818580.0  4.236797 -15.558192  16.683332         NaN\n",
       "29      9.0    2487.0  13787180.0  4.236797 -15.561426  16.671474         NaN"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture['val'] = agriculture['val'].fillna(0)\n",
    "agriculture.head(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>2356.0</td>\n",
       "      <td>1880419.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-15.379193</td>\n",
       "      <td>12.217335</td>\n",
       "      <td>93.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>398.0</td>\n",
       "      <td>18901.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-13.542333</td>\n",
       "      <td>15.381391</td>\n",
       "      <td>94.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>1683.0</td>\n",
       "      <td>329033.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.318076</td>\n",
       "      <td>13.362432</td>\n",
       "      <td>95.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>2407.0</td>\n",
       "      <td>854693.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-15.792235</td>\n",
       "      <td>12.125412</td>\n",
       "      <td>96.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>2444.0</td>\n",
       "      <td>1651475.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.292337</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>97.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>2442.0</td>\n",
       "      <td>2134537.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.459003</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>98.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>2451.0</td>\n",
       "      <td>426432.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.710119</td>\n",
       "      <td>12.131277</td>\n",
       "      <td>99.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>72.0</td>\n",
       "      <td>11654.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-14.042330</td>\n",
       "      <td>16.541962</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>2001.0</td>\n",
       "      <td>9111.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.125671</td>\n",
       "      <td>12.791977</td>\n",
       "      <td>101.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102</th>\n",
       "      <td>2344.0</td>\n",
       "      <td>2008851.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-16.458988</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>102.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>2213.0</td>\n",
       "      <td>202335.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.153987</td>\n",
       "      <td>12.457738</td>\n",
       "      <td>103.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>2437.0</td>\n",
       "      <td>822621.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.840583</td>\n",
       "      <td>12.164992</td>\n",
       "      <td>104.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>2397.0</td>\n",
       "      <td>337394.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.709006</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>105.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>2390.0</td>\n",
       "      <td>1125597.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.292337</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>106.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107</th>\n",
       "      <td>1736.0</td>\n",
       "      <td>835555.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.298256</td>\n",
       "      <td>13.287977</td>\n",
       "      <td>107.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>2.0</td>\n",
       "      <td>70993.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-15.374255</td>\n",
       "      <td>16.867753</td>\n",
       "      <td>108.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>2391.0</td>\n",
       "      <td>719387.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.209004</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>109.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>2502.0</td>\n",
       "      <td>557950.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.883042</td>\n",
       "      <td>12.077762</td>\n",
       "      <td>110.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>111</th>\n",
       "      <td>2387.0</td>\n",
       "      <td>950026.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.542336</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>111.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>112</th>\n",
       "      <td>2350.0</td>\n",
       "      <td>2447828.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-15.875657</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>112.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>7.0</td>\n",
       "      <td>901.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-14.875661</td>\n",
       "      <td>16.785873</td>\n",
       "      <td>113.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>2053.0</td>\n",
       "      <td>215844.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.209008</td>\n",
       "      <td>12.708644</td>\n",
       "      <td>114.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>115</th>\n",
       "      <td>2003.0</td>\n",
       "      <td>215284.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.292341</td>\n",
       "      <td>12.791977</td>\n",
       "      <td>115.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>2357.0</td>\n",
       "      <td>1024815.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-15.299521</td>\n",
       "      <td>12.233041</td>\n",
       "      <td>116.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>2501.0</td>\n",
       "      <td>407669.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-11.962580</td>\n",
       "      <td>12.072311</td>\n",
       "      <td>117.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>5.0</td>\n",
       "      <td>54822.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-15.375659</td>\n",
       "      <td>16.791961</td>\n",
       "      <td>118.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>2383.0</td>\n",
       "      <td>290657.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.873696</td>\n",
       "      <td>12.212042</td>\n",
       "      <td>119.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>2382.0</td>\n",
       "      <td>743827.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.955614</td>\n",
       "      <td>12.224018</td>\n",
       "      <td>120.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>2345.0</td>\n",
       "      <td>94605.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-16.375655</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>121.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>2443.0</td>\n",
       "      <td>1451706.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>122.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ID_spam   grid_val  ID_LC  val          x          y  Unnamed: 0\n",
       "93    2356.0  1880419.0    0.1  0.0 -15.379193  12.217335        93.0\n",
       "94     398.0    18901.0    0.1  0.0 -13.542333  15.381391        94.0\n",
       "95    1683.0   329033.0    0.1  0.0 -11.318076  13.362432        95.0\n",
       "96    2407.0   854693.0    0.1  0.0 -15.792235  12.125412        96.0\n",
       "97    2444.0  1651475.0    0.1  0.0 -12.292337  12.125313        97.0\n",
       "98    2442.0  2134537.0    0.1  0.0 -12.459003  12.125313        98.0\n",
       "99    2451.0   426432.0    0.1  0.0 -11.710119  12.131277        99.0\n",
       "100     72.0    11654.0    0.1  0.0 -14.042330  16.541962       100.0\n",
       "101   2001.0     9111.0    0.1  0.0 -12.125671  12.791977       101.0\n",
       "102   2344.0  2008851.0    0.1  0.0 -16.458988  12.208646       102.0\n",
       "103   2213.0   202335.0    0.1  0.0 -11.153987  12.457738       103.0\n",
       "104   2437.0   822621.0    0.1  0.0 -12.840583  12.164992       104.0\n",
       "105   2397.0   337394.0    0.1  0.0 -11.709006  12.208646       105.0\n",
       "106   2390.0  1125597.0    0.1  0.0 -12.292337  12.208646       106.0\n",
       "107   1736.0   835555.0    0.1  0.0 -11.298256  13.287977       107.0\n",
       "108      2.0    70993.0    0.1  0.0 -15.374255  16.867753       108.0\n",
       "109   2391.0   719387.0    0.1  0.0 -12.209004  12.208646       109.0\n",
       "110   2502.0   557950.0    0.1  0.0 -11.883042  12.077762       110.0\n",
       "111   2387.0   950026.0    0.1  0.0 -12.542336  12.208646       111.0\n",
       "112   2350.0  2447828.0    0.1  0.0 -15.875657  12.208646       112.0\n",
       "113      7.0      901.0    0.1  0.0 -14.875661  16.785873       113.0\n",
       "114   2053.0   215844.0    0.1  0.0 -11.209008  12.708644       114.0\n",
       "115   2003.0   215284.0    0.1  0.0 -11.292341  12.791977       115.0\n",
       "116   2357.0  1024815.0    0.1  0.0 -15.299521  12.233041       116.0\n",
       "117   2501.0   407669.0    0.1  0.0 -11.962580  12.072311       117.0\n",
       "118      5.0    54822.0    0.1  0.0 -15.375659  16.791961       118.0\n",
       "119   2383.0   290657.0    0.1  0.0 -12.873696  12.212042       119.0\n",
       "120   2382.0   743827.0    0.1  0.0 -12.955614  12.224018       120.0\n",
       "121   2345.0    94605.0    0.1  0.0 -16.375655  12.208646       121.0\n",
       "122   2443.0  1451706.0    0.1  0.0 -12.375670  12.125313       122.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture.tail(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13941114</th>\n",
       "      <td>113</td>\n",
       "      <td>7.0</td>\n",
       "      <td>901.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>901.0</td>\n",
       "      <td>-14.875661</td>\n",
       "      <td>16.785873</td>\n",
       "      <td>113.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941115</th>\n",
       "      <td>114</td>\n",
       "      <td>2053.0</td>\n",
       "      <td>215844.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>215844.0</td>\n",
       "      <td>-11.209008</td>\n",
       "      <td>12.708644</td>\n",
       "      <td>114.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941116</th>\n",
       "      <td>115</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>215284.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>215284.0</td>\n",
       "      <td>-11.292341</td>\n",
       "      <td>12.791977</td>\n",
       "      <td>115.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941117</th>\n",
       "      <td>116</td>\n",
       "      <td>2357.0</td>\n",
       "      <td>1024815.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1024815.0</td>\n",
       "      <td>-15.299521</td>\n",
       "      <td>12.233041</td>\n",
       "      <td>116.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941118</th>\n",
       "      <td>117</td>\n",
       "      <td>2501.0</td>\n",
       "      <td>407669.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>407669.0</td>\n",
       "      <td>-11.962580</td>\n",
       "      <td>12.072311</td>\n",
       "      <td>117.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941119</th>\n",
       "      <td>118</td>\n",
       "      <td>5.0</td>\n",
       "      <td>54822.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>54822.0</td>\n",
       "      <td>-15.375659</td>\n",
       "      <td>16.791961</td>\n",
       "      <td>118.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941120</th>\n",
       "      <td>119</td>\n",
       "      <td>2383.0</td>\n",
       "      <td>290657.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>290657.0</td>\n",
       "      <td>-12.873696</td>\n",
       "      <td>12.212042</td>\n",
       "      <td>119.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941121</th>\n",
       "      <td>120</td>\n",
       "      <td>2382.0</td>\n",
       "      <td>743827.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>743827.0</td>\n",
       "      <td>-12.955614</td>\n",
       "      <td>12.224018</td>\n",
       "      <td>120.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941122</th>\n",
       "      <td>121</td>\n",
       "      <td>2345.0</td>\n",
       "      <td>94605.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>94605.0</td>\n",
       "      <td>-16.375655</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>121.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941123</th>\n",
       "      <td>122</td>\n",
       "      <td>2443.0</td>\n",
       "      <td>1451706.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1451706.0</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>122.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          index  ID_spam   grid_val  ID_LC        val          x          y  \\\n",
       "13941114    113      7.0      901.0    0.1      901.0 -14.875661  16.785873   \n",
       "13941115    114   2053.0   215844.0    0.1   215844.0 -11.209008  12.708644   \n",
       "13941116    115   2003.0   215284.0    0.1   215284.0 -11.292341  12.791977   \n",
       "13941117    116   2357.0  1024815.0    0.1  1024815.0 -15.299521  12.233041   \n",
       "13941118    117   2501.0   407669.0    0.1   407669.0 -11.962580  12.072311   \n",
       "13941119    118      5.0    54822.0    0.1    54822.0 -15.375659  16.791961   \n",
       "13941120    119   2383.0   290657.0    0.1   290657.0 -12.873696  12.212042   \n",
       "13941121    120   2382.0   743827.0    0.1   743827.0 -12.955614  12.224018   \n",
       "13941122    121   2345.0    94605.0    0.1    94605.0 -16.375655  12.208646   \n",
       "13941123    122   2443.0  1451706.0    0.1  1451706.0 -12.375670  12.125313   \n",
       "\n",
       "          Unnamed: 0  \n",
       "13941114       113.0  \n",
       "13941115       114.0  \n",
       "13941116       115.0  \n",
       "13941117       116.0  \n",
       "13941118       117.0  \n",
       "13941119       118.0  \n",
       "13941120       119.0  \n",
       "13941121       120.0  \n",
       "13941122       121.0  \n",
       "13941123       122.0  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transfer grid values from spam to point values column. Should have done this via name change before appending.\n",
    "agriculture.reset_index(inplace=True)\n",
    "agriculture.loc[agriculture['val'] == 0, 'val'] = agriculture['grid_val']\n",
    "agriculture.tail(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787138.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.673091</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787238.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.581368</td>\n",
       "      <td>16.667162</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818609.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.553880</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818411.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.544717</td>\n",
       "      <td>16.691956</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787224.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571667</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818566.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.558192</td>\n",
       "      <td>16.683871</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818621.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.553880</td>\n",
       "      <td>16.681715</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787064.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559270</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>13</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818414.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.551185</td>\n",
       "      <td>16.691417</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>14</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787065.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.567894</td>\n",
       "      <td>16.676325</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>15</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818605.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.556036</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>16</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818520.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.525314</td>\n",
       "      <td>16.686566</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>17</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818604.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.556575</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>18</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818380.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.545795</td>\n",
       "      <td>16.693034</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>19</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818579.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.520463</td>\n",
       "      <td>16.683871</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    index  ID_spam  grid_val       ID_LC       val          x          y  \\\n",
       "0       0      9.0    2487.0  13787187.0  4.236797 -15.562504  16.670935   \n",
       "1       1      9.0    2487.0  13818608.0  4.236797 -15.554419  16.682254   \n",
       "2       2      9.0    2487.0  13787063.0  4.236797 -15.559809  16.676864   \n",
       "3       3      9.0    2487.0  13787188.0  4.236797 -15.561965  16.670935   \n",
       "4       4      9.0    2487.0  13787225.0  4.236797 -15.571128  16.668240   \n",
       "5       5      9.0    2487.0  13787138.0  4.236797 -15.561965  16.673091   \n",
       "6       6      9.0    2487.0  13787238.0  4.236797 -15.581368  16.667162   \n",
       "7       7      9.0    2487.0  13818609.0  4.236797 -15.553880  16.682254   \n",
       "8       8      9.0    2487.0  13818411.0  4.236797 -15.544717  16.691956   \n",
       "9       9      9.0    2487.0  13787224.0  4.236797 -15.571667  16.668240   \n",
       "10     10      9.0    2487.0  13818566.0  4.236797 -15.558192  16.683871   \n",
       "11     11      9.0    2487.0  13818621.0  4.236797 -15.553880  16.681715   \n",
       "12     12      9.0    2487.0  13787064.0  4.236797 -15.559270  16.676864   \n",
       "13     13      9.0    2487.0  13818414.0  4.236797 -15.551185  16.691417   \n",
       "14     14      9.0    2487.0  13787065.0  4.236797 -15.567894  16.676325   \n",
       "15     15      9.0    2487.0  13818605.0  4.236797 -15.556036  16.682254   \n",
       "16     16      9.0    2487.0  13818520.0  4.236797 -15.525314  16.686566   \n",
       "17     17      9.0    2487.0  13818604.0  4.236797 -15.556575  16.682254   \n",
       "18     18      9.0    2487.0  13818380.0  4.236797 -15.545795  16.693034   \n",
       "19     19      9.0    2487.0  13818579.0  4.236797 -15.520463  16.683871   \n",
       "\n",
       "    Unnamed: 0  \n",
       "0          NaN  \n",
       "1          NaN  \n",
       "2          NaN  \n",
       "3          NaN  \n",
       "4          NaN  \n",
       "5          NaN  \n",
       "6          NaN  \n",
       "7          NaN  \n",
       "8          NaN  \n",
       "9          NaN  \n",
       "10         NaN  \n",
       "11         NaN  \n",
       "12         NaN  \n",
       "13         NaN  \n",
       "14         NaN  \n",
       "15         NaN  \n",
       "16         NaN  \n",
       "17         NaN  \n",
       "18         NaN  \n",
       "19         NaN  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "agriculture.to_csv(os.path.join(out_pth, 'agriculture.csv')) # CSV circumvents some gdf issues later on.\n",
    "# Note that it will take a few min to write to file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Origins and destinations\n",
    "\n",
    "Measure distance from origin/destination to nearest node and save to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%% If starting new session, reload graph from file\n",
    "gTime = nx.read_gpickle(\"SEN-Cdrive/gTime.pickle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(358284990, 5217543379, {'osmid': 59618174, 'ref': 'D 523', 'name': 'D 523', 'highway': 'unclassified', 'oneway': False, 'length': 33.127, 'time': 2.3851440000000004, 'mode': 'drive'})\n"
     ]
    }
   ],
   "source": [
    "gn.example_edge(gTime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13941124 entries, 0 to 13941123\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Dtype   \n",
      "---  ------        -----   \n",
      " 0   Unnamed: 0    int64   \n",
      " 1   index         int64   \n",
      " 2   ID_spam       float64 \n",
      " 3   grid_val      float64 \n",
      " 4   ID_LC         float64 \n",
      " 5   val           float64 \n",
      " 6   x             float64 \n",
      " 7   y             float64 \n",
      " 8   Unnamed: 0.1  float64 \n",
      " 9   geometry      geometry\n",
      "dtypes: float64(7), geometry(1), int64(2)\n",
      "memory usage: 1.0 GB\n"
     ]
    }
   ],
   "source": [
    "# pandana_snap_c was giving an Index Error. Resetting the index didn't work.\n",
    "# Re-loading seemed to resolve whatever the issue was.\n",
    "agriculture = os.path.join(out_pth, \"agriculture.csv\")\n",
    "agriculture = pd.read_csv(agriculture)\n",
    "crs = \"EPSG:4326\"\n",
    "geometry = [Point(xy) for xy in zip(agriculture.x, agriculture.y)]\n",
    "agriculture = GeoDataFrame(agriculture, crs=crs, geometry=geometry) \n",
    "agriculture.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>index</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>geometry</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.56250 16.67094)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.55442 16.68225)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.55981 16.67686)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.56197 16.67094)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.57113 16.66824)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  index  ID_spam  grid_val       ID_LC       val          x  \\\n",
       "0           0      0      9.0    2487.0  13787187.0  4.236797 -15.562504   \n",
       "1           1      1      9.0    2487.0  13818608.0  4.236797 -15.554419   \n",
       "2           2      2      9.0    2487.0  13787063.0  4.236797 -15.559809   \n",
       "3           3      3      9.0    2487.0  13787188.0  4.236797 -15.561965   \n",
       "4           4      4      9.0    2487.0  13787225.0  4.236797 -15.571128   \n",
       "\n",
       "           y  Unnamed: 0.1                    geometry  \n",
       "0  16.670935           NaN  POINT (-15.56250 16.67094)  \n",
       "1  16.682254           NaN  POINT (-15.55442 16.68225)  \n",
       "2  16.676864           NaN  POINT (-15.55981 16.67686)  \n",
       "3  16.670935           NaN  POINT (-15.56197 16.67094)  \n",
       "4  16.668240           NaN  POINT (-15.57113 16.66824)  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Geographic 2D CRS: EPSG:4326>\n",
       "Name: WGS 84\n",
       "Axis Info [ellipsoidal]:\n",
       "- Lat[north]: Geodetic latitude (degree)\n",
       "- Lon[east]: Geodetic longitude (degree)\n",
       "Area of Use:\n",
       "- name: World.\n",
       "- bounds: (-180.0, -90.0, 180.0, 90.0)\n",
       "Datum: World Geodetic System 1984 ensemble\n",
       "- Ellipsoid: WGS 84\n",
       "- Prime Meridian: Greenwich"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agriculture.crs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start: Thu Dec 23 12:03:59 2021\n",
      "\n",
      "\n",
      "end: Thu Dec 23 12:27:21 2021\n",
      "\n",
      "--- processing complete\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>index</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>geometry</th>\n",
       "      <th>NN</th>\n",
       "      <th>NN_dist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.56250 16.67094)</td>\n",
       "      <td>3656618126</td>\n",
       "      <td>5113.670395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.55442 16.68225)</td>\n",
       "      <td>3656617769</td>\n",
       "      <td>5632.260552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.55981 16.67686)</td>\n",
       "      <td>3656617751</td>\n",
       "      <td>5453.608825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.56197 16.67094)</td>\n",
       "      <td>3656618126</td>\n",
       "      <td>5075.881917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.57113 16.66824)</td>\n",
       "      <td>3656618126</td>\n",
       "      <td>5568.582130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941119</th>\n",
       "      <td>13941119</td>\n",
       "      <td>118</td>\n",
       "      <td>5.0</td>\n",
       "      <td>54822.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>5.482200e+04</td>\n",
       "      <td>-15.375659</td>\n",
       "      <td>16.791961</td>\n",
       "      <td>118.0</td>\n",
       "      <td>POINT (-15.37566 16.79196)</td>\n",
       "      <td>3651042393</td>\n",
       "      <td>4280.425149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941120</th>\n",
       "      <td>13941120</td>\n",
       "      <td>119</td>\n",
       "      <td>2383.0</td>\n",
       "      <td>290657.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>2.906570e+05</td>\n",
       "      <td>-12.873696</td>\n",
       "      <td>12.212042</td>\n",
       "      <td>119.0</td>\n",
       "      <td>POINT (-12.87370 12.21204)</td>\n",
       "      <td>4818951142</td>\n",
       "      <td>9019.155622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941121</th>\n",
       "      <td>13941121</td>\n",
       "      <td>120</td>\n",
       "      <td>2382.0</td>\n",
       "      <td>743827.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>7.438270e+05</td>\n",
       "      <td>-12.955614</td>\n",
       "      <td>12.224018</td>\n",
       "      <td>120.0</td>\n",
       "      <td>POINT (-12.95561 12.22402)</td>\n",
       "      <td>4618077921</td>\n",
       "      <td>10786.551522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941122</th>\n",
       "      <td>13941122</td>\n",
       "      <td>121</td>\n",
       "      <td>2345.0</td>\n",
       "      <td>94605.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>9.460500e+04</td>\n",
       "      <td>-16.375655</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>121.0</td>\n",
       "      <td>POINT (-16.37565 12.20865)</td>\n",
       "      <td>3507831510</td>\n",
       "      <td>2881.578459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941123</th>\n",
       "      <td>13941123</td>\n",
       "      <td>122</td>\n",
       "      <td>2443.0</td>\n",
       "      <td>1451706.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.451706e+06</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>122.0</td>\n",
       "      <td>POINT (-12.37567 12.12531)</td>\n",
       "      <td>3882487056</td>\n",
       "      <td>398.111940</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13941124 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          Unnamed: 0  index  ID_spam   grid_val       ID_LC           val  \\\n",
       "0                  0      0      9.0     2487.0  13787187.0  4.236797e+00   \n",
       "1                  1      1      9.0     2487.0  13818608.0  4.236797e+00   \n",
       "2                  2      2      9.0     2487.0  13787063.0  4.236797e+00   \n",
       "3                  3      3      9.0     2487.0  13787188.0  4.236797e+00   \n",
       "4                  4      4      9.0     2487.0  13787225.0  4.236797e+00   \n",
       "...              ...    ...      ...        ...         ...           ...   \n",
       "13941119    13941119    118      5.0    54822.0         0.1  5.482200e+04   \n",
       "13941120    13941120    119   2383.0   290657.0         0.1  2.906570e+05   \n",
       "13941121    13941121    120   2382.0   743827.0         0.1  7.438270e+05   \n",
       "13941122    13941122    121   2345.0    94605.0         0.1  9.460500e+04   \n",
       "13941123    13941123    122   2443.0  1451706.0         0.1  1.451706e+06   \n",
       "\n",
       "                  x          y  Unnamed: 0.1                    geometry  \\\n",
       "0        -15.562504  16.670935           NaN  POINT (-15.56250 16.67094)   \n",
       "1        -15.554419  16.682254           NaN  POINT (-15.55442 16.68225)   \n",
       "2        -15.559809  16.676864           NaN  POINT (-15.55981 16.67686)   \n",
       "3        -15.561965  16.670935           NaN  POINT (-15.56197 16.67094)   \n",
       "4        -15.571128  16.668240           NaN  POINT (-15.57113 16.66824)   \n",
       "...             ...        ...           ...                         ...   \n",
       "13941119 -15.375659  16.791961         118.0  POINT (-15.37566 16.79196)   \n",
       "13941120 -12.873696  12.212042         119.0  POINT (-12.87370 12.21204)   \n",
       "13941121 -12.955614  12.224018         120.0  POINT (-12.95561 12.22402)   \n",
       "13941122 -16.375655  12.208646         121.0  POINT (-16.37565 12.20865)   \n",
       "13941123 -12.375670  12.125313         122.0  POINT (-12.37567 12.12531)   \n",
       "\n",
       "                  NN       NN_dist  \n",
       "0         3656618126   5113.670395  \n",
       "1         3656617769   5632.260552  \n",
       "2         3656617751   5453.608825  \n",
       "3         3656618126   5075.881917  \n",
       "4         3656618126   5568.582130  \n",
       "...              ...           ...  \n",
       "13941119  3651042393   4280.425149  \n",
       "13941120  4818951142   9019.155622  \n",
       "13941121  4618077921  10786.551522  \n",
       "13941122  3507831510   2881.578459  \n",
       "13941123  3882487056    398.111940  \n",
       "\n",
       "[13941124 rows x 12 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('start: %s\\n' % time.ctime())\n",
    "ag_snap = gn.pandana_snap_c(gTime, agriculture, source_crs = 'epsg:4326', target_crs = 'epsg:31028', add_dist_to_node_col = True)\n",
    "ag_snap.to_csv('SEN-Cdrive/outputs/ag_snap.csv', index=True)\n",
    "print('\\nend: %s' % time.ctime())\n",
    "print('\\n--- processing complete')\n",
    "ag_snap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create travel time values for the road nodes nearest to each service.\n",
    "\n",
    "Using calculate_OD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If starting a new session, load from file.\n",
    "HDurban_snap = os.path.join(out_pth, \"HDurban_snap.csv\")\n",
    "HDurban_snap = pd.read_csv(HDurban_snap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ag_snap = os.path.join(out_pth, \"ag_snap.csv\")\n",
    "ag_snap = pd.read_csv(ag_snap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "gTime = nx.read_gpickle(\"SEN-Cdrive/outputs/gTime.pickle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We only need to find the origin-destination pairs for nodes closest to the origins and services,\n",
    "# and some nodes will be the nearest for more than one service (and definitely for multiple origins).\n",
    "origins = list(ag_snap.NN.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_HDurban = list(HDurban_snap.NN.unique()) \n",
    "dests = list_HDurban\n",
    "\n",
    "# If more than one set of destinations, you can combine the sets to go into a single OD matrix using this code:\n",
    "# list_ssa = list(ssa_snap.NN.unique()) \n",
    "# destslist = list_HDurban + list_ssa\n",
    "# dests = list(set(destslist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "625851"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(origins) # 23108 unique nearest nodes (average of 5 origins per node)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "58"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dests) # 314 unique nearest nodes. This is only 5 less than the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "fail_value = 999999999 # If there is no shortest path, the OD pair will be assigned the fail value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(358284990, 5217543379, {'osmid': 59618174, 'ref': 'D 523', 'name': 'D 523', 'highway': 'unclassified', 'oneway': False, 'length': 33.127, 'time': 2.3851440000000004, 'mode': 'drive'})\n",
      "(358284990, 1888282175, {'osmid': 178482063, 'highway': 'tertiary', 'oneway': False, 'length': 12.832, 'time': 0.76992, 'mode': 'drive'})\n",
      "(358284990, 5329792467, {'osmid': 178482063, 'highway': 'tertiary', 'oneway': False, 'length': 48.781, 'time': 2.92686, 'mode': 'drive'})\n",
      "(358284993, 1888282575, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 126.577, 'time': 7.59462, 'mode': 'drive'})\n",
      "(358284993, 1888198886, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 120.578, 'time': 7.23468, 'mode': 'drive'})\n",
      "(358284994, 6424643487, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 46.582, 'time': 2.79492, 'mode': 'drive'})\n",
      "(358284994, 1888282618, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 53.36, 'time': 3.2016, 'mode': 'drive'})\n",
      "(358284995, 1888282666, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 73.745, 'time': 4.4247000000000005, 'mode': 'drive'})\n",
      "(358284995, 1888282669, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 63.931, 'time': 3.8358600000000003, 'mode': 'drive'})\n",
      "(358284997, 1888282780, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 175.337, 'time': 10.52022, 'mode': 'drive'})\n",
      "(358284997, 1888282790, {'osmid': 178470940, 'highway': 'tertiary', 'oneway': False, 'length': 88.008, 'time': 5.280479999999999, 'mode': 'drive'})\n"
     ]
    }
   ],
   "source": [
    "gn.example_edge(gTime, 11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "OD = gn.calculate_OD(gTime, origins, dests, fail_value, weight = 'time')\n",
    "# This a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "OD_df = pd.DataFrame(OD, index = origins, columns = dests)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 625851 entries, 3656618126 to 3882487056\n",
      "Data columns (total 58 columns):\n",
      " #   Column      Non-Null Count   Dtype  \n",
      "---  ------      --------------   -----  \n",
      " 0   6058226279  625851 non-null  float64\n",
      " 1   6029307183  625851 non-null  float64\n",
      " 2   4998093094  625851 non-null  float64\n",
      " 3   2201506815  625851 non-null  float64\n",
      " 4   3474499811  625851 non-null  float64\n",
      " 5   1697006012  625851 non-null  float64\n",
      " 6   1901689169  625851 non-null  float64\n",
      " 7   6032060028  625851 non-null  float64\n",
      " 8   6040927878  625851 non-null  float64\n",
      " 9   3449495495  625851 non-null  float64\n",
      " 10  3990543961  625851 non-null  float64\n",
      " 11  8972391475  625851 non-null  float64\n",
      " 12  3418418812  625851 non-null  float64\n",
      " 13  1983641803  625851 non-null  float64\n",
      " 14  6014451367  625851 non-null  float64\n",
      " 15  6027163276  625851 non-null  float64\n",
      " 16  2833577858  625851 non-null  float64\n",
      " 17  4656728818  625851 non-null  float64\n",
      " 18  6045659373  625851 non-null  float64\n",
      " 19  5528866190  625851 non-null  float64\n",
      " 20  6027939517  625851 non-null  float64\n",
      " 21  6031093257  625851 non-null  float64\n",
      " 22  1883155712  625851 non-null  float64\n",
      " 23  6026417971  625851 non-null  float64\n",
      " 24  6031245697  625851 non-null  float64\n",
      " 25  7493947593  625851 non-null  float64\n",
      " 26  1859359090  625851 non-null  float64\n",
      " 27  6053816462  625851 non-null  float64\n",
      " 28  6027530676  625851 non-null  float64\n",
      " 29  2988435647  625851 non-null  float64\n",
      " 30  1332801944  625851 non-null  float64\n",
      " 31  6053825179  625851 non-null  float64\n",
      " 32  2988435792  625851 non-null  float64\n",
      " 33  3036877534  625851 non-null  float64\n",
      " 34  6056472223  625851 non-null  float64\n",
      " 35  7460241128  625851 non-null  float64\n",
      " 36  6028595154  625851 non-null  float64\n",
      " 37  4053739078  625851 non-null  float64\n",
      " 38  4071261250  625851 non-null  float64\n",
      " 39  6014815762  625851 non-null  float64\n",
      " 40  6024430122  625851 non-null  float64\n",
      " 41  8229317581  625851 non-null  float64\n",
      " 42  6027834438  625851 non-null  float64\n",
      " 43  6024894515  625851 non-null  float64\n",
      " 44  6024894530  625851 non-null  float64\n",
      " 45  6024894513  625851 non-null  float64\n",
      " 46  6031993937  625851 non-null  float64\n",
      " 47  8200298999  625851 non-null  float64\n",
      " 48  1968458114  625851 non-null  float64\n",
      " 49  1936967272  625851 non-null  float64\n",
      " 50  3496518021  625851 non-null  float64\n",
      " 51  6027615161  625851 non-null  float64\n",
      " 52  6027276892  625851 non-null  float64\n",
      " 53  6041228287  625851 non-null  float64\n",
      " 54  5536661253  625851 non-null  float64\n",
      " 55  7357630367  625851 non-null  float64\n",
      " 56  8178147277  625851 non-null  float64\n",
      " 57  6026834850  625851 non-null  float64\n",
      "dtypes: float64(58)\n",
      "memory usage: 281.7 MB\n"
     ]
    }
   ],
   "source": [
    "OD_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>6058226279</th>\n",
       "      <th>6029307183</th>\n",
       "      <th>4998093094</th>\n",
       "      <th>2201506815</th>\n",
       "      <th>3474499811</th>\n",
       "      <th>1697006012</th>\n",
       "      <th>1901689169</th>\n",
       "      <th>6032060028</th>\n",
       "      <th>6040927878</th>\n",
       "      <th>3449495495</th>\n",
       "      <th>...</th>\n",
       "      <th>1968458114</th>\n",
       "      <th>1936967272</th>\n",
       "      <th>3496518021</th>\n",
       "      <th>6027615161</th>\n",
       "      <th>6027276892</th>\n",
       "      <th>6041228287</th>\n",
       "      <th>5536661253</th>\n",
       "      <th>7357630367</th>\n",
       "      <th>8178147277</th>\n",
       "      <th>6026834850</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3691541254</th>\n",
       "      <td>32667.287962</td>\n",
       "      <td>24304.383961</td>\n",
       "      <td>33719.340799</td>\n",
       "      <td>28806.555570</td>\n",
       "      <td>25967.007870</td>\n",
       "      <td>35154.292086</td>\n",
       "      <td>36467.971926</td>\n",
       "      <td>35440.340053</td>\n",
       "      <td>31701.636060</td>\n",
       "      <td>31867.242606</td>\n",
       "      <td>...</td>\n",
       "      <td>37413.787711</td>\n",
       "      <td>40744.568312</td>\n",
       "      <td>16581.282252</td>\n",
       "      <td>16570.282542</td>\n",
       "      <td>33979.333880</td>\n",
       "      <td>36278.401638</td>\n",
       "      <td>36333.506280</td>\n",
       "      <td>36644.209122</td>\n",
       "      <td>32081.918197</td>\n",
       "      <td>32394.376391</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5376874047</th>\n",
       "      <td>7908.832112</td>\n",
       "      <td>6934.580049</td>\n",
       "      <td>22409.257730</td>\n",
       "      <td>17496.472500</td>\n",
       "      <td>17645.623638</td>\n",
       "      <td>23844.209017</td>\n",
       "      <td>25157.888856</td>\n",
       "      <td>24130.256983</td>\n",
       "      <td>20391.552990</td>\n",
       "      <td>22413.580490</td>\n",
       "      <td>...</td>\n",
       "      <td>27627.202833</td>\n",
       "      <td>30790.094698</td>\n",
       "      <td>16615.685029</td>\n",
       "      <td>16637.134924</td>\n",
       "      <td>22669.250811</td>\n",
       "      <td>24968.318569</td>\n",
       "      <td>25023.423211</td>\n",
       "      <td>25334.126053</td>\n",
       "      <td>22280.968551</td>\n",
       "      <td>23002.540878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3651042393</th>\n",
       "      <td>36502.461371</td>\n",
       "      <td>35422.517484</td>\n",
       "      <td>22589.899624</td>\n",
       "      <td>24947.833976</td>\n",
       "      <td>26372.879713</td>\n",
       "      <td>20464.283865</td>\n",
       "      <td>22763.696706</td>\n",
       "      <td>19448.397677</td>\n",
       "      <td>22052.458466</td>\n",
       "      <td>20515.640119</td>\n",
       "      <td>...</td>\n",
       "      <td>15279.003582</td>\n",
       "      <td>11638.548276</td>\n",
       "      <td>35723.238333</td>\n",
       "      <td>35973.784863</td>\n",
       "      <td>22680.903888</td>\n",
       "      <td>22475.895024</td>\n",
       "      <td>22444.199192</td>\n",
       "      <td>22438.946802</td>\n",
       "      <td>20638.326724</td>\n",
       "      <td>20702.081530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4618077921</th>\n",
       "      <td>23418.745092</td>\n",
       "      <td>15052.180971</td>\n",
       "      <td>29383.372930</td>\n",
       "      <td>24470.587701</td>\n",
       "      <td>21631.040001</td>\n",
       "      <td>30818.324217</td>\n",
       "      <td>32132.004057</td>\n",
       "      <td>31104.372184</td>\n",
       "      <td>27365.668191</td>\n",
       "      <td>27531.274737</td>\n",
       "      <td>...</td>\n",
       "      <td>33077.819842</td>\n",
       "      <td>36408.600443</td>\n",
       "      <td>12253.070703</td>\n",
       "      <td>12274.520598</td>\n",
       "      <td>29643.366011</td>\n",
       "      <td>31942.433769</td>\n",
       "      <td>31997.538411</td>\n",
       "      <td>32308.241253</td>\n",
       "      <td>27745.950328</td>\n",
       "      <td>28058.408522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882487056</th>\n",
       "      <td>30679.315465</td>\n",
       "      <td>22316.411464</td>\n",
       "      <td>31731.368302</td>\n",
       "      <td>26818.583073</td>\n",
       "      <td>23979.035373</td>\n",
       "      <td>33166.319589</td>\n",
       "      <td>34479.999429</td>\n",
       "      <td>33452.367556</td>\n",
       "      <td>29713.663563</td>\n",
       "      <td>29879.270109</td>\n",
       "      <td>...</td>\n",
       "      <td>35425.815214</td>\n",
       "      <td>38756.595815</td>\n",
       "      <td>14593.309755</td>\n",
       "      <td>14582.310045</td>\n",
       "      <td>31991.361383</td>\n",
       "      <td>34290.429141</td>\n",
       "      <td>34345.533783</td>\n",
       "      <td>34656.236625</td>\n",
       "      <td>30093.945700</td>\n",
       "      <td>30406.403894</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              6058226279    6029307183    4998093094    2201506815  \\\n",
       "3691541254  32667.287962  24304.383961  33719.340799  28806.555570   \n",
       "5376874047   7908.832112   6934.580049  22409.257730  17496.472500   \n",
       "3651042393  36502.461371  35422.517484  22589.899624  24947.833976   \n",
       "4618077921  23418.745092  15052.180971  29383.372930  24470.587701   \n",
       "3882487056  30679.315465  22316.411464  31731.368302  26818.583073   \n",
       "\n",
       "              3474499811    1697006012    1901689169    6032060028  \\\n",
       "3691541254  25967.007870  35154.292086  36467.971926  35440.340053   \n",
       "5376874047  17645.623638  23844.209017  25157.888856  24130.256983   \n",
       "3651042393  26372.879713  20464.283865  22763.696706  19448.397677   \n",
       "4618077921  21631.040001  30818.324217  32132.004057  31104.372184   \n",
       "3882487056  23979.035373  33166.319589  34479.999429  33452.367556   \n",
       "\n",
       "              6040927878    3449495495  ...    1968458114    1936967272  \\\n",
       "3691541254  31701.636060  31867.242606  ...  37413.787711  40744.568312   \n",
       "5376874047  20391.552990  22413.580490  ...  27627.202833  30790.094698   \n",
       "3651042393  22052.458466  20515.640119  ...  15279.003582  11638.548276   \n",
       "4618077921  27365.668191  27531.274737  ...  33077.819842  36408.600443   \n",
       "3882487056  29713.663563  29879.270109  ...  35425.815214  38756.595815   \n",
       "\n",
       "              3496518021    6027615161    6027276892    6041228287  \\\n",
       "3691541254  16581.282252  16570.282542  33979.333880  36278.401638   \n",
       "5376874047  16615.685029  16637.134924  22669.250811  24968.318569   \n",
       "3651042393  35723.238333  35973.784863  22680.903888  22475.895024   \n",
       "4618077921  12253.070703  12274.520598  29643.366011  31942.433769   \n",
       "3882487056  14593.309755  14582.310045  31991.361383  34290.429141   \n",
       "\n",
       "              5536661253    7357630367    8178147277    6026834850  \n",
       "3691541254  36333.506280  36644.209122  32081.918197  32394.376391  \n",
       "5376874047  25023.423211  25334.126053  22280.968551  23002.540878  \n",
       "3651042393  22444.199192  22438.946802  20638.326724  20702.081530  \n",
       "4618077921  31997.538411  32308.241253  27745.950328  28058.408522  \n",
       "3882487056  34345.533783  34656.236625  30093.945700  30406.403894  \n",
       "\n",
       "[5 rows x 58 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "OD_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to minutes and save to file.\n",
    "OD_min = OD_df[OD_df <fail_value] / 60\n",
    "OD_min.to_csv(os.path.join(out_pth, 'fromagriculture/OD.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>6058226279</th>\n",
       "      <th>6029307183</th>\n",
       "      <th>4998093094</th>\n",
       "      <th>2201506815</th>\n",
       "      <th>3474499811</th>\n",
       "      <th>1697006012</th>\n",
       "      <th>1901689169</th>\n",
       "      <th>6032060028</th>\n",
       "      <th>6040927878</th>\n",
       "      <th>3449495495</th>\n",
       "      <th>...</th>\n",
       "      <th>1968458114</th>\n",
       "      <th>1936967272</th>\n",
       "      <th>3496518021</th>\n",
       "      <th>6027615161</th>\n",
       "      <th>6027276892</th>\n",
       "      <th>6041228287</th>\n",
       "      <th>5536661253</th>\n",
       "      <th>7357630367</th>\n",
       "      <th>8178147277</th>\n",
       "      <th>6026834850</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5072506110</th>\n",
       "      <td>520.447595</td>\n",
       "      <td>381.065861</td>\n",
       "      <td>537.981809</td>\n",
       "      <td>456.102055</td>\n",
       "      <td>408.776260</td>\n",
       "      <td>561.897663</td>\n",
       "      <td>583.792327</td>\n",
       "      <td>566.665129</td>\n",
       "      <td>504.353396</td>\n",
       "      <td>507.113505</td>\n",
       "      <td>...</td>\n",
       "      <td>599.555924</td>\n",
       "      <td>655.068934</td>\n",
       "      <td>252.347499</td>\n",
       "      <td>252.164171</td>\n",
       "      <td>542.315027</td>\n",
       "      <td>580.632823</td>\n",
       "      <td>581.551233</td>\n",
       "      <td>586.729614</td>\n",
       "      <td>510.691432</td>\n",
       "      <td>515.899068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1860634594</th>\n",
       "      <td>712.079413</td>\n",
       "      <td>694.080348</td>\n",
       "      <td>480.203384</td>\n",
       "      <td>519.502290</td>\n",
       "      <td>543.253052</td>\n",
       "      <td>444.776454</td>\n",
       "      <td>483.100002</td>\n",
       "      <td>427.845018</td>\n",
       "      <td>471.246031</td>\n",
       "      <td>445.632392</td>\n",
       "      <td>...</td>\n",
       "      <td>358.355116</td>\n",
       "      <td>297.680861</td>\n",
       "      <td>699.092362</td>\n",
       "      <td>703.268138</td>\n",
       "      <td>481.720122</td>\n",
       "      <td>478.303307</td>\n",
       "      <td>477.775043</td>\n",
       "      <td>477.687503</td>\n",
       "      <td>447.677169</td>\n",
       "      <td>448.739749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5111800060</th>\n",
       "      <td>444.547341</td>\n",
       "      <td>305.165607</td>\n",
       "      <td>462.081555</td>\n",
       "      <td>380.201801</td>\n",
       "      <td>332.876006</td>\n",
       "      <td>485.997410</td>\n",
       "      <td>507.892074</td>\n",
       "      <td>490.764876</td>\n",
       "      <td>428.453142</td>\n",
       "      <td>431.213252</td>\n",
       "      <td>...</td>\n",
       "      <td>523.655670</td>\n",
       "      <td>579.168680</td>\n",
       "      <td>176.447246</td>\n",
       "      <td>176.263917</td>\n",
       "      <td>466.414773</td>\n",
       "      <td>504.732569</td>\n",
       "      <td>505.650979</td>\n",
       "      <td>510.829360</td>\n",
       "      <td>434.791178</td>\n",
       "      <td>439.998815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3507831609</th>\n",
       "      <td>83.755636</td>\n",
       "      <td>209.586049</td>\n",
       "      <td>357.743419</td>\n",
       "      <td>275.863666</td>\n",
       "      <td>278.349518</td>\n",
       "      <td>381.659274</td>\n",
       "      <td>403.553938</td>\n",
       "      <td>386.426740</td>\n",
       "      <td>324.115007</td>\n",
       "      <td>357.815465</td>\n",
       "      <td>...</td>\n",
       "      <td>444.709171</td>\n",
       "      <td>497.424036</td>\n",
       "      <td>370.937799</td>\n",
       "      <td>371.295297</td>\n",
       "      <td>362.076637</td>\n",
       "      <td>400.394433</td>\n",
       "      <td>401.312844</td>\n",
       "      <td>406.491225</td>\n",
       "      <td>355.605266</td>\n",
       "      <td>367.631472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3109029200</th>\n",
       "      <td>586.630365</td>\n",
       "      <td>447.248632</td>\n",
       "      <td>604.164579</td>\n",
       "      <td>522.284825</td>\n",
       "      <td>474.959030</td>\n",
       "      <td>628.080434</td>\n",
       "      <td>649.975098</td>\n",
       "      <td>632.847900</td>\n",
       "      <td>570.536167</td>\n",
       "      <td>573.296276</td>\n",
       "      <td>...</td>\n",
       "      <td>665.738694</td>\n",
       "      <td>721.251704</td>\n",
       "      <td>318.530270</td>\n",
       "      <td>318.346941</td>\n",
       "      <td>608.497797</td>\n",
       "      <td>646.815593</td>\n",
       "      <td>647.734004</td>\n",
       "      <td>652.912384</td>\n",
       "      <td>576.874202</td>\n",
       "      <td>582.081839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2535965098</th>\n",
       "      <td>520.499464</td>\n",
       "      <td>381.117730</td>\n",
       "      <td>538.033678</td>\n",
       "      <td>456.153924</td>\n",
       "      <td>408.828129</td>\n",
       "      <td>561.949532</td>\n",
       "      <td>583.844196</td>\n",
       "      <td>566.716998</td>\n",
       "      <td>504.405265</td>\n",
       "      <td>507.165374</td>\n",
       "      <td>...</td>\n",
       "      <td>599.607793</td>\n",
       "      <td>655.120803</td>\n",
       "      <td>252.399368</td>\n",
       "      <td>252.216040</td>\n",
       "      <td>542.366896</td>\n",
       "      <td>580.684692</td>\n",
       "      <td>581.603102</td>\n",
       "      <td>586.781483</td>\n",
       "      <td>510.743301</td>\n",
       "      <td>515.950937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1888510171</th>\n",
       "      <td>499.414740</td>\n",
       "      <td>360.033007</td>\n",
       "      <td>516.948954</td>\n",
       "      <td>435.069201</td>\n",
       "      <td>387.743406</td>\n",
       "      <td>540.864809</td>\n",
       "      <td>562.759473</td>\n",
       "      <td>545.632275</td>\n",
       "      <td>483.320542</td>\n",
       "      <td>486.080651</td>\n",
       "      <td>...</td>\n",
       "      <td>578.523070</td>\n",
       "      <td>634.036080</td>\n",
       "      <td>231.314645</td>\n",
       "      <td>231.131317</td>\n",
       "      <td>521.282172</td>\n",
       "      <td>559.599968</td>\n",
       "      <td>560.518379</td>\n",
       "      <td>565.696760</td>\n",
       "      <td>489.658578</td>\n",
       "      <td>494.866214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3166661162</th>\n",
       "      <td>557.505539</td>\n",
       "      <td>418.123806</td>\n",
       "      <td>575.039753</td>\n",
       "      <td>493.159999</td>\n",
       "      <td>445.834204</td>\n",
       "      <td>598.955608</td>\n",
       "      <td>620.850272</td>\n",
       "      <td>603.723074</td>\n",
       "      <td>541.411341</td>\n",
       "      <td>544.171450</td>\n",
       "      <td>...</td>\n",
       "      <td>636.613868</td>\n",
       "      <td>674.867498</td>\n",
       "      <td>289.405444</td>\n",
       "      <td>289.222115</td>\n",
       "      <td>579.372971</td>\n",
       "      <td>617.690767</td>\n",
       "      <td>618.609178</td>\n",
       "      <td>623.787558</td>\n",
       "      <td>547.749376</td>\n",
       "      <td>552.957013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3651042501</th>\n",
       "      <td>621.904659</td>\n",
       "      <td>603.905594</td>\n",
       "      <td>390.028629</td>\n",
       "      <td>429.327535</td>\n",
       "      <td>453.078298</td>\n",
       "      <td>354.601700</td>\n",
       "      <td>392.925247</td>\n",
       "      <td>337.670264</td>\n",
       "      <td>381.071277</td>\n",
       "      <td>355.457638</td>\n",
       "      <td>...</td>\n",
       "      <td>268.180362</td>\n",
       "      <td>207.506107</td>\n",
       "      <td>608.917608</td>\n",
       "      <td>613.093383</td>\n",
       "      <td>391.545367</td>\n",
       "      <td>388.128553</td>\n",
       "      <td>387.600289</td>\n",
       "      <td>387.512749</td>\n",
       "      <td>357.502414</td>\n",
       "      <td>358.564995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1888509577</th>\n",
       "      <td>509.488486</td>\n",
       "      <td>370.106753</td>\n",
       "      <td>527.022700</td>\n",
       "      <td>445.142946</td>\n",
       "      <td>397.817151</td>\n",
       "      <td>550.938555</td>\n",
       "      <td>572.833219</td>\n",
       "      <td>555.706021</td>\n",
       "      <td>493.394288</td>\n",
       "      <td>496.154397</td>\n",
       "      <td>...</td>\n",
       "      <td>588.596815</td>\n",
       "      <td>644.109825</td>\n",
       "      <td>241.388391</td>\n",
       "      <td>241.205062</td>\n",
       "      <td>531.355918</td>\n",
       "      <td>569.673714</td>\n",
       "      <td>570.592125</td>\n",
       "      <td>575.770505</td>\n",
       "      <td>499.732323</td>\n",
       "      <td>504.939960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3086849624</th>\n",
       "      <td>525.355616</td>\n",
       "      <td>385.973882</td>\n",
       "      <td>542.889830</td>\n",
       "      <td>461.010076</td>\n",
       "      <td>413.684281</td>\n",
       "      <td>566.805684</td>\n",
       "      <td>588.700348</td>\n",
       "      <td>571.573150</td>\n",
       "      <td>509.261417</td>\n",
       "      <td>512.021526</td>\n",
       "      <td>...</td>\n",
       "      <td>604.463945</td>\n",
       "      <td>659.976955</td>\n",
       "      <td>257.255520</td>\n",
       "      <td>257.072192</td>\n",
       "      <td>547.223048</td>\n",
       "      <td>585.540844</td>\n",
       "      <td>586.459254</td>\n",
       "      <td>591.637635</td>\n",
       "      <td>515.599453</td>\n",
       "      <td>520.807089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4929043683</th>\n",
       "      <td>541.438560</td>\n",
       "      <td>402.056826</td>\n",
       "      <td>558.972774</td>\n",
       "      <td>477.093020</td>\n",
       "      <td>429.767225</td>\n",
       "      <td>582.888629</td>\n",
       "      <td>604.783293</td>\n",
       "      <td>587.656095</td>\n",
       "      <td>525.344361</td>\n",
       "      <td>528.104471</td>\n",
       "      <td>...</td>\n",
       "      <td>620.546889</td>\n",
       "      <td>676.059899</td>\n",
       "      <td>273.338465</td>\n",
       "      <td>273.155136</td>\n",
       "      <td>563.305992</td>\n",
       "      <td>601.623788</td>\n",
       "      <td>602.542198</td>\n",
       "      <td>607.720579</td>\n",
       "      <td>531.682397</td>\n",
       "      <td>536.890034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8605349340</th>\n",
       "      <td>89.193106</td>\n",
       "      <td>146.942609</td>\n",
       "      <td>363.180890</td>\n",
       "      <td>281.301136</td>\n",
       "      <td>283.786988</td>\n",
       "      <td>387.096745</td>\n",
       "      <td>408.991408</td>\n",
       "      <td>391.864211</td>\n",
       "      <td>329.552477</td>\n",
       "      <td>363.252936</td>\n",
       "      <td>...</td>\n",
       "      <td>450.146641</td>\n",
       "      <td>502.861506</td>\n",
       "      <td>308.294359</td>\n",
       "      <td>308.651857</td>\n",
       "      <td>367.514108</td>\n",
       "      <td>405.831904</td>\n",
       "      <td>406.750314</td>\n",
       "      <td>411.928695</td>\n",
       "      <td>361.042737</td>\n",
       "      <td>373.068942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6323434997</th>\n",
       "      <td>631.520070</td>\n",
       "      <td>613.521005</td>\n",
       "      <td>399.644041</td>\n",
       "      <td>438.942947</td>\n",
       "      <td>462.693709</td>\n",
       "      <td>364.217112</td>\n",
       "      <td>402.540659</td>\n",
       "      <td>347.285675</td>\n",
       "      <td>390.686688</td>\n",
       "      <td>365.073049</td>\n",
       "      <td>...</td>\n",
       "      <td>277.795774</td>\n",
       "      <td>217.121519</td>\n",
       "      <td>618.533020</td>\n",
       "      <td>622.708795</td>\n",
       "      <td>401.160779</td>\n",
       "      <td>397.743964</td>\n",
       "      <td>397.215701</td>\n",
       "      <td>397.128161</td>\n",
       "      <td>367.117826</td>\n",
       "      <td>368.180406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3338441152</th>\n",
       "      <td>553.145904</td>\n",
       "      <td>413.764171</td>\n",
       "      <td>570.680118</td>\n",
       "      <td>488.800364</td>\n",
       "      <td>441.474569</td>\n",
       "      <td>594.595973</td>\n",
       "      <td>616.490637</td>\n",
       "      <td>599.363439</td>\n",
       "      <td>537.051706</td>\n",
       "      <td>539.811815</td>\n",
       "      <td>...</td>\n",
       "      <td>632.254233</td>\n",
       "      <td>687.767243</td>\n",
       "      <td>285.045809</td>\n",
       "      <td>284.862480</td>\n",
       "      <td>575.013336</td>\n",
       "      <td>613.331132</td>\n",
       "      <td>614.249543</td>\n",
       "      <td>619.427923</td>\n",
       "      <td>543.389741</td>\n",
       "      <td>548.597378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3691541254</th>\n",
       "      <td>544.454799</td>\n",
       "      <td>405.073066</td>\n",
       "      <td>561.989013</td>\n",
       "      <td>480.109260</td>\n",
       "      <td>432.783464</td>\n",
       "      <td>585.904868</td>\n",
       "      <td>607.799532</td>\n",
       "      <td>590.672334</td>\n",
       "      <td>528.360601</td>\n",
       "      <td>531.120710</td>\n",
       "      <td>...</td>\n",
       "      <td>623.563129</td>\n",
       "      <td>679.076139</td>\n",
       "      <td>276.354704</td>\n",
       "      <td>276.171376</td>\n",
       "      <td>566.322231</td>\n",
       "      <td>604.640027</td>\n",
       "      <td>605.558438</td>\n",
       "      <td>610.736819</td>\n",
       "      <td>534.698637</td>\n",
       "      <td>539.906273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5376874047</th>\n",
       "      <td>131.813869</td>\n",
       "      <td>115.576334</td>\n",
       "      <td>373.487629</td>\n",
       "      <td>291.607875</td>\n",
       "      <td>294.093727</td>\n",
       "      <td>397.403484</td>\n",
       "      <td>419.298148</td>\n",
       "      <td>402.170950</td>\n",
       "      <td>339.859217</td>\n",
       "      <td>373.559675</td>\n",
       "      <td>...</td>\n",
       "      <td>460.453381</td>\n",
       "      <td>513.168245</td>\n",
       "      <td>276.928084</td>\n",
       "      <td>277.285582</td>\n",
       "      <td>377.820847</td>\n",
       "      <td>416.138643</td>\n",
       "      <td>417.057054</td>\n",
       "      <td>422.235434</td>\n",
       "      <td>371.349476</td>\n",
       "      <td>383.375681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3651042393</th>\n",
       "      <td>608.374356</td>\n",
       "      <td>590.375291</td>\n",
       "      <td>376.498327</td>\n",
       "      <td>415.797233</td>\n",
       "      <td>439.547995</td>\n",
       "      <td>341.071398</td>\n",
       "      <td>379.394945</td>\n",
       "      <td>324.139961</td>\n",
       "      <td>367.540974</td>\n",
       "      <td>341.927335</td>\n",
       "      <td>...</td>\n",
       "      <td>254.650060</td>\n",
       "      <td>193.975805</td>\n",
       "      <td>595.387306</td>\n",
       "      <td>599.563081</td>\n",
       "      <td>378.015065</td>\n",
       "      <td>374.598250</td>\n",
       "      <td>374.069987</td>\n",
       "      <td>373.982447</td>\n",
       "      <td>343.972112</td>\n",
       "      <td>345.034692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4618077921</th>\n",
       "      <td>390.312418</td>\n",
       "      <td>250.869683</td>\n",
       "      <td>489.722882</td>\n",
       "      <td>407.843128</td>\n",
       "      <td>360.517333</td>\n",
       "      <td>513.638737</td>\n",
       "      <td>535.533401</td>\n",
       "      <td>518.406203</td>\n",
       "      <td>456.094470</td>\n",
       "      <td>458.854579</td>\n",
       "      <td>...</td>\n",
       "      <td>551.296997</td>\n",
       "      <td>606.810007</td>\n",
       "      <td>204.217845</td>\n",
       "      <td>204.575343</td>\n",
       "      <td>494.056100</td>\n",
       "      <td>532.373896</td>\n",
       "      <td>533.292307</td>\n",
       "      <td>538.470688</td>\n",
       "      <td>462.432505</td>\n",
       "      <td>467.640142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882487056</th>\n",
       "      <td>511.321924</td>\n",
       "      <td>371.940191</td>\n",
       "      <td>528.856138</td>\n",
       "      <td>446.976385</td>\n",
       "      <td>399.650590</td>\n",
       "      <td>552.771993</td>\n",
       "      <td>574.666657</td>\n",
       "      <td>557.539459</td>\n",
       "      <td>495.227726</td>\n",
       "      <td>497.987835</td>\n",
       "      <td>...</td>\n",
       "      <td>590.430254</td>\n",
       "      <td>645.943264</td>\n",
       "      <td>243.221829</td>\n",
       "      <td>243.038501</td>\n",
       "      <td>533.189356</td>\n",
       "      <td>571.507152</td>\n",
       "      <td>572.425563</td>\n",
       "      <td>577.603944</td>\n",
       "      <td>501.565762</td>\n",
       "      <td>506.773398</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            6058226279  6029307183  4998093094  2201506815  3474499811  \\\n",
       "5072506110  520.447595  381.065861  537.981809  456.102055  408.776260   \n",
       "1860634594  712.079413  694.080348  480.203384  519.502290  543.253052   \n",
       "5111800060  444.547341  305.165607  462.081555  380.201801  332.876006   \n",
       "3507831609   83.755636  209.586049  357.743419  275.863666  278.349518   \n",
       "3109029200  586.630365  447.248632  604.164579  522.284825  474.959030   \n",
       "2535965098  520.499464  381.117730  538.033678  456.153924  408.828129   \n",
       "1888510171  499.414740  360.033007  516.948954  435.069201  387.743406   \n",
       "3166661162  557.505539  418.123806  575.039753  493.159999  445.834204   \n",
       "3651042501  621.904659  603.905594  390.028629  429.327535  453.078298   \n",
       "1888509577  509.488486  370.106753  527.022700  445.142946  397.817151   \n",
       "3086849624  525.355616  385.973882  542.889830  461.010076  413.684281   \n",
       "4929043683  541.438560  402.056826  558.972774  477.093020  429.767225   \n",
       "8605349340   89.193106  146.942609  363.180890  281.301136  283.786988   \n",
       "6323434997  631.520070  613.521005  399.644041  438.942947  462.693709   \n",
       "3338441152  553.145904  413.764171  570.680118  488.800364  441.474569   \n",
       "3691541254  544.454799  405.073066  561.989013  480.109260  432.783464   \n",
       "5376874047  131.813869  115.576334  373.487629  291.607875  294.093727   \n",
       "3651042393  608.374356  590.375291  376.498327  415.797233  439.547995   \n",
       "4618077921  390.312418  250.869683  489.722882  407.843128  360.517333   \n",
       "3882487056  511.321924  371.940191  528.856138  446.976385  399.650590   \n",
       "\n",
       "            1697006012  1901689169  6032060028  6040927878  3449495495  ...  \\\n",
       "5072506110  561.897663  583.792327  566.665129  504.353396  507.113505  ...   \n",
       "1860634594  444.776454  483.100002  427.845018  471.246031  445.632392  ...   \n",
       "5111800060  485.997410  507.892074  490.764876  428.453142  431.213252  ...   \n",
       "3507831609  381.659274  403.553938  386.426740  324.115007  357.815465  ...   \n",
       "3109029200  628.080434  649.975098  632.847900  570.536167  573.296276  ...   \n",
       "2535965098  561.949532  583.844196  566.716998  504.405265  507.165374  ...   \n",
       "1888510171  540.864809  562.759473  545.632275  483.320542  486.080651  ...   \n",
       "3166661162  598.955608  620.850272  603.723074  541.411341  544.171450  ...   \n",
       "3651042501  354.601700  392.925247  337.670264  381.071277  355.457638  ...   \n",
       "1888509577  550.938555  572.833219  555.706021  493.394288  496.154397  ...   \n",
       "3086849624  566.805684  588.700348  571.573150  509.261417  512.021526  ...   \n",
       "4929043683  582.888629  604.783293  587.656095  525.344361  528.104471  ...   \n",
       "8605349340  387.096745  408.991408  391.864211  329.552477  363.252936  ...   \n",
       "6323434997  364.217112  402.540659  347.285675  390.686688  365.073049  ...   \n",
       "3338441152  594.595973  616.490637  599.363439  537.051706  539.811815  ...   \n",
       "3691541254  585.904868  607.799532  590.672334  528.360601  531.120710  ...   \n",
       "5376874047  397.403484  419.298148  402.170950  339.859217  373.559675  ...   \n",
       "3651042393  341.071398  379.394945  324.139961  367.540974  341.927335  ...   \n",
       "4618077921  513.638737  535.533401  518.406203  456.094470  458.854579  ...   \n",
       "3882487056  552.771993  574.666657  557.539459  495.227726  497.987835  ...   \n",
       "\n",
       "            1968458114  1936967272  3496518021  6027615161  6027276892  \\\n",
       "5072506110  599.555924  655.068934  252.347499  252.164171  542.315027   \n",
       "1860634594  358.355116  297.680861  699.092362  703.268138  481.720122   \n",
       "5111800060  523.655670  579.168680  176.447246  176.263917  466.414773   \n",
       "3507831609  444.709171  497.424036  370.937799  371.295297  362.076637   \n",
       "3109029200  665.738694  721.251704  318.530270  318.346941  608.497797   \n",
       "2535965098  599.607793  655.120803  252.399368  252.216040  542.366896   \n",
       "1888510171  578.523070  634.036080  231.314645  231.131317  521.282172   \n",
       "3166661162  636.613868  674.867498  289.405444  289.222115  579.372971   \n",
       "3651042501  268.180362  207.506107  608.917608  613.093383  391.545367   \n",
       "1888509577  588.596815  644.109825  241.388391  241.205062  531.355918   \n",
       "3086849624  604.463945  659.976955  257.255520  257.072192  547.223048   \n",
       "4929043683  620.546889  676.059899  273.338465  273.155136  563.305992   \n",
       "8605349340  450.146641  502.861506  308.294359  308.651857  367.514108   \n",
       "6323434997  277.795774  217.121519  618.533020  622.708795  401.160779   \n",
       "3338441152  632.254233  687.767243  285.045809  284.862480  575.013336   \n",
       "3691541254  623.563129  679.076139  276.354704  276.171376  566.322231   \n",
       "5376874047  460.453381  513.168245  276.928084  277.285582  377.820847   \n",
       "3651042393  254.650060  193.975805  595.387306  599.563081  378.015065   \n",
       "4618077921  551.296997  606.810007  204.217845  204.575343  494.056100   \n",
       "3882487056  590.430254  645.943264  243.221829  243.038501  533.189356   \n",
       "\n",
       "            6041228287  5536661253  7357630367  8178147277  6026834850  \n",
       "5072506110  580.632823  581.551233  586.729614  510.691432  515.899068  \n",
       "1860634594  478.303307  477.775043  477.687503  447.677169  448.739749  \n",
       "5111800060  504.732569  505.650979  510.829360  434.791178  439.998815  \n",
       "3507831609  400.394433  401.312844  406.491225  355.605266  367.631472  \n",
       "3109029200  646.815593  647.734004  652.912384  576.874202  582.081839  \n",
       "2535965098  580.684692  581.603102  586.781483  510.743301  515.950937  \n",
       "1888510171  559.599968  560.518379  565.696760  489.658578  494.866214  \n",
       "3166661162  617.690767  618.609178  623.787558  547.749376  552.957013  \n",
       "3651042501  388.128553  387.600289  387.512749  357.502414  358.564995  \n",
       "1888509577  569.673714  570.592125  575.770505  499.732323  504.939960  \n",
       "3086849624  585.540844  586.459254  591.637635  515.599453  520.807089  \n",
       "4929043683  601.623788  602.542198  607.720579  531.682397  536.890034  \n",
       "8605349340  405.831904  406.750314  411.928695  361.042737  373.068942  \n",
       "6323434997  397.743964  397.215701  397.128161  367.117826  368.180406  \n",
       "3338441152  613.331132  614.249543  619.427923  543.389741  548.597378  \n",
       "3691541254  604.640027  605.558438  610.736819  534.698637  539.906273  \n",
       "5376874047  416.138643  417.057054  422.235434  371.349476  383.375681  \n",
       "3651042393  374.598250  374.069987  373.982447  343.972112  345.034692  \n",
       "4618077921  532.373896  533.292307  538.470688  462.432505  467.640142  \n",
       "3882487056  571.507152  572.425563  577.603944  501.565762  506.773398  \n",
       "\n",
       "[20 rows x 58 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "OD_min.tail(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter 1st nearest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check each file to make sure nearest neighbor column is named correctly. If not, rename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reload from file even if already loaded. Quickest way to ensure NN is a column rather than only the index.\n",
    "OD = os.path.join(out_pth, \"fromagriculture/OD.csv\")\n",
    "OD = pd.read_csv(OD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>6058226279</th>\n",
       "      <th>6029307183</th>\n",
       "      <th>4998093094</th>\n",
       "      <th>2201506815</th>\n",
       "      <th>3474499811</th>\n",
       "      <th>1697006012</th>\n",
       "      <th>1901689169</th>\n",
       "      <th>6032060028</th>\n",
       "      <th>6040927878</th>\n",
       "      <th>...</th>\n",
       "      <th>1968458114</th>\n",
       "      <th>1936967272</th>\n",
       "      <th>3496518021</th>\n",
       "      <th>6027615161</th>\n",
       "      <th>6027276892</th>\n",
       "      <th>6041228287</th>\n",
       "      <th>5536661253</th>\n",
       "      <th>7357630367</th>\n",
       "      <th>8178147277</th>\n",
       "      <th>6026834850</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3656618126</td>\n",
       "      <td>577.901536</td>\n",
       "      <td>559.902472</td>\n",
       "      <td>346.025507</td>\n",
       "      <td>385.324413</td>\n",
       "      <td>409.075175</td>\n",
       "      <td>310.598578</td>\n",
       "      <td>348.922125</td>\n",
       "      <td>293.667142</td>\n",
       "      <td>337.068155</td>\n",
       "      <td>...</td>\n",
       "      <td>224.177240</td>\n",
       "      <td>163.502985</td>\n",
       "      <td>564.914486</td>\n",
       "      <td>569.090261</td>\n",
       "      <td>347.542245</td>\n",
       "      <td>344.125431</td>\n",
       "      <td>343.597167</td>\n",
       "      <td>343.509627</td>\n",
       "      <td>313.499292</td>\n",
       "      <td>314.561872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3656617769</td>\n",
       "      <td>578.309363</td>\n",
       "      <td>560.310299</td>\n",
       "      <td>346.433334</td>\n",
       "      <td>385.732240</td>\n",
       "      <td>409.483002</td>\n",
       "      <td>311.006405</td>\n",
       "      <td>349.329952</td>\n",
       "      <td>294.074969</td>\n",
       "      <td>337.475982</td>\n",
       "      <td>...</td>\n",
       "      <td>224.585067</td>\n",
       "      <td>163.910812</td>\n",
       "      <td>565.322313</td>\n",
       "      <td>569.498088</td>\n",
       "      <td>347.950072</td>\n",
       "      <td>344.533258</td>\n",
       "      <td>344.004994</td>\n",
       "      <td>343.917454</td>\n",
       "      <td>313.907119</td>\n",
       "      <td>314.969699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3656617751</td>\n",
       "      <td>577.966765</td>\n",
       "      <td>559.967701</td>\n",
       "      <td>346.090736</td>\n",
       "      <td>385.389642</td>\n",
       "      <td>409.140404</td>\n",
       "      <td>310.663807</td>\n",
       "      <td>348.987354</td>\n",
       "      <td>293.732371</td>\n",
       "      <td>337.133384</td>\n",
       "      <td>...</td>\n",
       "      <td>224.242469</td>\n",
       "      <td>163.568214</td>\n",
       "      <td>564.979715</td>\n",
       "      <td>569.155490</td>\n",
       "      <td>347.607474</td>\n",
       "      <td>344.190660</td>\n",
       "      <td>343.662396</td>\n",
       "      <td>343.574856</td>\n",
       "      <td>313.564521</td>\n",
       "      <td>314.627101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4303444021</td>\n",
       "      <td>571.873871</td>\n",
       "      <td>553.874806</td>\n",
       "      <td>339.997842</td>\n",
       "      <td>379.296748</td>\n",
       "      <td>403.047510</td>\n",
       "      <td>304.570912</td>\n",
       "      <td>342.894460</td>\n",
       "      <td>287.639476</td>\n",
       "      <td>331.040489</td>\n",
       "      <td>...</td>\n",
       "      <td>218.149574</td>\n",
       "      <td>157.475319</td>\n",
       "      <td>558.886820</td>\n",
       "      <td>563.062596</td>\n",
       "      <td>341.514579</td>\n",
       "      <td>338.097765</td>\n",
       "      <td>337.569501</td>\n",
       "      <td>337.481961</td>\n",
       "      <td>307.471627</td>\n",
       "      <td>308.534207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3656617804</td>\n",
       "      <td>578.333584</td>\n",
       "      <td>560.334520</td>\n",
       "      <td>346.457555</td>\n",
       "      <td>385.756461</td>\n",
       "      <td>409.507223</td>\n",
       "      <td>311.030626</td>\n",
       "      <td>349.354173</td>\n",
       "      <td>294.099190</td>\n",
       "      <td>337.500203</td>\n",
       "      <td>...</td>\n",
       "      <td>224.609288</td>\n",
       "      <td>163.935033</td>\n",
       "      <td>565.346534</td>\n",
       "      <td>569.522309</td>\n",
       "      <td>347.974293</td>\n",
       "      <td>344.557479</td>\n",
       "      <td>344.029215</td>\n",
       "      <td>343.941675</td>\n",
       "      <td>313.931340</td>\n",
       "      <td>314.993920</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625846</th>\n",
       "      <td>3691541254</td>\n",
       "      <td>544.454799</td>\n",
       "      <td>405.073066</td>\n",
       "      <td>561.989013</td>\n",
       "      <td>480.109260</td>\n",
       "      <td>432.783464</td>\n",
       "      <td>585.904868</td>\n",
       "      <td>607.799532</td>\n",
       "      <td>590.672334</td>\n",
       "      <td>528.360601</td>\n",
       "      <td>...</td>\n",
       "      <td>623.563129</td>\n",
       "      <td>679.076139</td>\n",
       "      <td>276.354704</td>\n",
       "      <td>276.171376</td>\n",
       "      <td>566.322231</td>\n",
       "      <td>604.640027</td>\n",
       "      <td>605.558438</td>\n",
       "      <td>610.736819</td>\n",
       "      <td>534.698637</td>\n",
       "      <td>539.906273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625847</th>\n",
       "      <td>5376874047</td>\n",
       "      <td>131.813869</td>\n",
       "      <td>115.576334</td>\n",
       "      <td>373.487629</td>\n",
       "      <td>291.607875</td>\n",
       "      <td>294.093727</td>\n",
       "      <td>397.403484</td>\n",
       "      <td>419.298148</td>\n",
       "      <td>402.170950</td>\n",
       "      <td>339.859217</td>\n",
       "      <td>...</td>\n",
       "      <td>460.453381</td>\n",
       "      <td>513.168245</td>\n",
       "      <td>276.928084</td>\n",
       "      <td>277.285582</td>\n",
       "      <td>377.820847</td>\n",
       "      <td>416.138643</td>\n",
       "      <td>417.057054</td>\n",
       "      <td>422.235434</td>\n",
       "      <td>371.349476</td>\n",
       "      <td>383.375681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625848</th>\n",
       "      <td>3651042393</td>\n",
       "      <td>608.374356</td>\n",
       "      <td>590.375291</td>\n",
       "      <td>376.498327</td>\n",
       "      <td>415.797233</td>\n",
       "      <td>439.547995</td>\n",
       "      <td>341.071398</td>\n",
       "      <td>379.394945</td>\n",
       "      <td>324.139961</td>\n",
       "      <td>367.540974</td>\n",
       "      <td>...</td>\n",
       "      <td>254.650060</td>\n",
       "      <td>193.975805</td>\n",
       "      <td>595.387306</td>\n",
       "      <td>599.563081</td>\n",
       "      <td>378.015065</td>\n",
       "      <td>374.598250</td>\n",
       "      <td>374.069987</td>\n",
       "      <td>373.982447</td>\n",
       "      <td>343.972112</td>\n",
       "      <td>345.034692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625849</th>\n",
       "      <td>4618077921</td>\n",
       "      <td>390.312418</td>\n",
       "      <td>250.869683</td>\n",
       "      <td>489.722882</td>\n",
       "      <td>407.843128</td>\n",
       "      <td>360.517333</td>\n",
       "      <td>513.638737</td>\n",
       "      <td>535.533401</td>\n",
       "      <td>518.406203</td>\n",
       "      <td>456.094470</td>\n",
       "      <td>...</td>\n",
       "      <td>551.296997</td>\n",
       "      <td>606.810007</td>\n",
       "      <td>204.217845</td>\n",
       "      <td>204.575343</td>\n",
       "      <td>494.056100</td>\n",
       "      <td>532.373896</td>\n",
       "      <td>533.292307</td>\n",
       "      <td>538.470688</td>\n",
       "      <td>462.432505</td>\n",
       "      <td>467.640142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>625850</th>\n",
       "      <td>3882487056</td>\n",
       "      <td>511.321924</td>\n",
       "      <td>371.940191</td>\n",
       "      <td>528.856138</td>\n",
       "      <td>446.976385</td>\n",
       "      <td>399.650590</td>\n",
       "      <td>552.771993</td>\n",
       "      <td>574.666657</td>\n",
       "      <td>557.539459</td>\n",
       "      <td>495.227726</td>\n",
       "      <td>...</td>\n",
       "      <td>590.430254</td>\n",
       "      <td>645.943264</td>\n",
       "      <td>243.221829</td>\n",
       "      <td>243.038501</td>\n",
       "      <td>533.189356</td>\n",
       "      <td>571.507152</td>\n",
       "      <td>572.425563</td>\n",
       "      <td>577.603944</td>\n",
       "      <td>501.565762</td>\n",
       "      <td>506.773398</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>625851 rows × 59 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Unnamed: 0  6058226279  6029307183  4998093094  2201506815  \\\n",
       "0       3656618126  577.901536  559.902472  346.025507  385.324413   \n",
       "1       3656617769  578.309363  560.310299  346.433334  385.732240   \n",
       "2       3656617751  577.966765  559.967701  346.090736  385.389642   \n",
       "3       4303444021  571.873871  553.874806  339.997842  379.296748   \n",
       "4       3656617804  578.333584  560.334520  346.457555  385.756461   \n",
       "...            ...         ...         ...         ...         ...   \n",
       "625846  3691541254  544.454799  405.073066  561.989013  480.109260   \n",
       "625847  5376874047  131.813869  115.576334  373.487629  291.607875   \n",
       "625848  3651042393  608.374356  590.375291  376.498327  415.797233   \n",
       "625849  4618077921  390.312418  250.869683  489.722882  407.843128   \n",
       "625850  3882487056  511.321924  371.940191  528.856138  446.976385   \n",
       "\n",
       "        3474499811  1697006012  1901689169  6032060028  6040927878  ...  \\\n",
       "0       409.075175  310.598578  348.922125  293.667142  337.068155  ...   \n",
       "1       409.483002  311.006405  349.329952  294.074969  337.475982  ...   \n",
       "2       409.140404  310.663807  348.987354  293.732371  337.133384  ...   \n",
       "3       403.047510  304.570912  342.894460  287.639476  331.040489  ...   \n",
       "4       409.507223  311.030626  349.354173  294.099190  337.500203  ...   \n",
       "...            ...         ...         ...         ...         ...  ...   \n",
       "625846  432.783464  585.904868  607.799532  590.672334  528.360601  ...   \n",
       "625847  294.093727  397.403484  419.298148  402.170950  339.859217  ...   \n",
       "625848  439.547995  341.071398  379.394945  324.139961  367.540974  ...   \n",
       "625849  360.517333  513.638737  535.533401  518.406203  456.094470  ...   \n",
       "625850  399.650590  552.771993  574.666657  557.539459  495.227726  ...   \n",
       "\n",
       "        1968458114  1936967272  3496518021  6027615161  6027276892  \\\n",
       "0       224.177240  163.502985  564.914486  569.090261  347.542245   \n",
       "1       224.585067  163.910812  565.322313  569.498088  347.950072   \n",
       "2       224.242469  163.568214  564.979715  569.155490  347.607474   \n",
       "3       218.149574  157.475319  558.886820  563.062596  341.514579   \n",
       "4       224.609288  163.935033  565.346534  569.522309  347.974293   \n",
       "...            ...         ...         ...         ...         ...   \n",
       "625846  623.563129  679.076139  276.354704  276.171376  566.322231   \n",
       "625847  460.453381  513.168245  276.928084  277.285582  377.820847   \n",
       "625848  254.650060  193.975805  595.387306  599.563081  378.015065   \n",
       "625849  551.296997  606.810007  204.217845  204.575343  494.056100   \n",
       "625850  590.430254  645.943264  243.221829  243.038501  533.189356   \n",
       "\n",
       "        6041228287  5536661253  7357630367  8178147277  6026834850  \n",
       "0       344.125431  343.597167  343.509627  313.499292  314.561872  \n",
       "1       344.533258  344.004994  343.917454  313.907119  314.969699  \n",
       "2       344.190660  343.662396  343.574856  313.564521  314.627101  \n",
       "3       338.097765  337.569501  337.481961  307.471627  308.534207  \n",
       "4       344.557479  344.029215  343.941675  313.931340  314.993920  \n",
       "...            ...         ...         ...         ...         ...  \n",
       "625846  604.640027  605.558438  610.736819  534.698637  539.906273  \n",
       "625847  416.138643  417.057054  422.235434  371.349476  383.375681  \n",
       "625848  374.598250  374.069987  373.982447  343.972112  345.034692  \n",
       "625849  532.373896  533.292307  538.470688  462.432505  467.640142  \n",
       "625850  571.507152  572.425563  577.603944  501.565762  506.773398  \n",
       "\n",
       "[625851 rows x 59 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "OD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "OD.rename(columns={'Unnamed: 0': 'NN'}, inplace=True) # Repeat for each OD set, if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Find first, second, and third nearest destination for each origin node. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "fail_value = 999999999"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NN</th>\n",
       "      <th>HDurban1</th>\n",
       "      <th>HDurban2</th>\n",
       "      <th>HDurban3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3656618126</td>\n",
       "      <td>163.502985</td>\n",
       "      <td>164.457268</td>\n",
       "      <td>170.140815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3656617769</td>\n",
       "      <td>163.910812</td>\n",
       "      <td>164.865095</td>\n",
       "      <td>170.548642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3656617751</td>\n",
       "      <td>163.568214</td>\n",
       "      <td>164.522497</td>\n",
       "      <td>170.206044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4303444021</td>\n",
       "      <td>157.475319</td>\n",
       "      <td>158.429603</td>\n",
       "      <td>164.113150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3656617804</td>\n",
       "      <td>163.935033</td>\n",
       "      <td>164.889316</td>\n",
       "      <td>170.572863</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           NN    HDurban1    HDurban2    HDurban3\n",
       "0  3656618126  163.502985  164.457268  170.140815\n",
       "1  3656617769  163.910812  164.865095  170.548642\n",
       "2  3656617751  163.568214  164.522497  170.206044\n",
       "3  4303444021  157.475319  158.429603  164.113150\n",
       "4  3656617804  163.935033  164.889316  170.572863"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Since we only have one destination set, renaming the OD object for quick find/replace on this section of code.\n",
    "OD_HDurban = OD\n",
    "\n",
    "# Nearest\n",
    "OD_HDurban[\"HDurban1\"] = 0\n",
    "sub = OD_HDurban.iloc[:,1:-1] # Filtering out the newly created field and the node ID column. (\"include everything between column 0 and the last column\")\n",
    "OD_HDurban[\"HDurban1\"] = sub.min(axis=1) # Default is axis=0, meaning min value of each column selected. We want min of each row.\n",
    "HDurban1 = OD_HDurban[['NN', 'HDurban1']] # Remove unnecessary OD values.\n",
    "HDurban1.to_csv(os.path.join(out_pth, 'fromagriculture/ag-HDurban1.csv'))\n",
    "\n",
    "\n",
    "# Second nearest\n",
    "dupes = OD_HDurban.apply(pd.Series.duplicated, axis = 1, keep=False) # If a number is repeated within a row, value is True. If not, False.\n",
    "# The first time this is done, there should be two True values per row, unless any POIs are equidistant.\n",
    "dupes = OD_HDurban.where(~dupes, fail_value) # For any value that appears more than once in its row, it is replaced with the fail_value.\n",
    "\n",
    "OD_HDurban[\"HDurban2\"] = 0\n",
    "Dsub = dupes.iloc[:,1:] # Filtering out the node ID column. No need to filter 1st nearest as its new \"dupes\" value is too high to be caught.\n",
    "OD_HDurban[\"HDurban2\"] = Dsub.min(axis=1) \n",
    "HDurban2 = OD_HDurban.loc[:,['NN', 'HDurban2']] \n",
    "HDurban2.to_csv(os.path.join(out_pth, 'fromagriculture/ag-HDurban2.csv'))\n",
    "\n",
    "\n",
    "# Third nearest\n",
    "dupes = OD_HDurban.apply(pd.Series.duplicated, axis = 1, keep=False)\n",
    "# Since this includes both first and second nearest columns, there should be four True values per row, unless POIs are equidistant.\n",
    "dupes = OD_HDurban.where(~dupes, fail_value)\n",
    " \n",
    "OD_HDurban[\"HDurban3\"] = 0\n",
    "Dsub = dupes.iloc[:,1:] # Filtering out the node ID column.\n",
    "OD_HDurban[\"HDurban3\"] = Dsub.min(axis=1)\n",
    "HDurban3 = OD_HDurban.loc[:,['NN', 'HDurban3']]\n",
    "HDurban3.to_csv(os.path.join(out_pth, 'fromagriculture/ag-HDurban3.csv'))\n",
    "\n",
    "\n",
    "# Combine and write to file\n",
    "HDurban_all = OD_HDurban.loc[:,['NN', 'HDurban1', 'HDurban2', 'HDurban3']]\n",
    "HDurban_all.to_csv(os.path.join(out_pth, 'fromagriculture/ag-HDurban_all.csv'))\n",
    "HDurban_all.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Join back to georeferenced _snap file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\wb527163\\.conda\\envs\\geo\\lib\\site-packages\\IPython\\core\\interactiveshell.py:3444: DtypeWarning: Columns (9,10) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  exec(code_obj, self.user_global_ns, self.user_ns)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>ID_lc</th>\n",
       "      <th>geometry</th>\n",
       "      <th>index__spam</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>val</th>\n",
       "      <th>geom_UTM</th>\n",
       "      <th>geom_WGS84</th>\n",
       "      <th>lc_sum</th>\n",
       "      <th>spam_V</th>\n",
       "      <th>NN</th>\n",
       "      <th>NN_dist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-15.990057</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>1.0</td>\n",
       "      <td>POINT (-15.9900569732419 16.6937073347074)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5343951929</td>\n",
       "      <td>14607.791436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-15.989787</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>2.0</td>\n",
       "      <td>POINT (-15.9897874786566 16.6937073347074)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5343951929</td>\n",
       "      <td>14628.808767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-15.989518</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>3.0</td>\n",
       "      <td>POINT (-15.9895179840714 16.6937073347074)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5343951929</td>\n",
       "      <td>14649.852284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>-15.925378</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>4.0</td>\n",
       "      <td>POINT (-15.9253782727816 16.6937073347074)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3639403839</td>\n",
       "      <td>14029.850649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>-15.925109</td>\n",
       "      <td>16.693707</td>\n",
       "      <td>5.0</td>\n",
       "      <td>POINT (-15.9251087781964 16.6937073347074)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3639403839</td>\n",
       "      <td>14018.013147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7657060</th>\n",
       "      <td>7657060</td>\n",
       "      <td>2061</td>\n",
       "      <td>-11.459007</td>\n",
       "      <td>12.041980</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-11.459007 12.04198)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2507.0</td>\n",
       "      <td>411859.0</td>\n",
       "      <td>MULTIPOLYGON (((890065.2102591199 1329046.5648...</td>\n",
       "      <td>POINT (-11.45900694020285 12.041979526954448)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>inf</td>\n",
       "      <td>2241013982</td>\n",
       "      <td>14274.586682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7657061</th>\n",
       "      <td>7657061</td>\n",
       "      <td>2062</td>\n",
       "      <td>-11.375674</td>\n",
       "      <td>12.041980</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-11.375674 12.04198)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2508.0</td>\n",
       "      <td>497200.0</td>\n",
       "      <td>MULTIPOLYGON (((899153.5281643758 1329166.2613...</td>\n",
       "      <td>POINT (-11.375673926651558 12.041979526573865)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>inf</td>\n",
       "      <td>6662833878</td>\n",
       "      <td>13597.325940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7657062</th>\n",
       "      <td>7657062</td>\n",
       "      <td>2063</td>\n",
       "      <td>-11.292341</td>\n",
       "      <td>12.041980</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-11.292341 12.04198)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2509.0</td>\n",
       "      <td>217938.0</td>\n",
       "      <td>MULTIPOLYGON (((908242.6248140114 1329288.7282...</td>\n",
       "      <td>POINT (-11.292340913098172 12.041979526183944)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>inf</td>\n",
       "      <td>6662833878</td>\n",
       "      <td>14994.531096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7657063</th>\n",
       "      <td>7657063</td>\n",
       "      <td>2064</td>\n",
       "      <td>-11.209008</td>\n",
       "      <td>12.041980</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-11.209008 12.04198)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2510.0</td>\n",
       "      <td>201222.0</td>\n",
       "      <td>MULTIPOLYGON (((917332.5179180377 1329413.9667...</td>\n",
       "      <td>POINT (-11.209007899992315 12.041979525784605)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>inf</td>\n",
       "      <td>2217989971</td>\n",
       "      <td>20260.745415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7657064</th>\n",
       "      <td>7657064</td>\n",
       "      <td>2065</td>\n",
       "      <td>-11.125675</td>\n",
       "      <td>12.041980</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-11.125675 12.04198)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2511.0</td>\n",
       "      <td>944829.0</td>\n",
       "      <td>MULTIPOLYGON (((917332.5179180377 1329413.9667...</td>\n",
       "      <td>POINT (-11.125674886884132 12.04197952537604)</td>\n",
       "      <td>0.0</td>\n",
       "      <td>inf</td>\n",
       "      <td>3344553248</td>\n",
       "      <td>24408.027959</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7657065 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0  Unnamed: 0.1          x          y  ID_lc  \\\n",
       "0                 0             0 -15.990057  16.693707    1.0   \n",
       "1                 1             1 -15.989787  16.693707    2.0   \n",
       "2                 2             2 -15.989518  16.693707    3.0   \n",
       "3                 3             3 -15.925378  16.693707    4.0   \n",
       "4                 4             4 -15.925109  16.693707    5.0   \n",
       "...             ...           ...        ...        ...    ...   \n",
       "7657060     7657060          2061 -11.459007  12.041980    NaN   \n",
       "7657061     7657061          2062 -11.375674  12.041980    NaN   \n",
       "7657062     7657062          2063 -11.292341  12.041980    NaN   \n",
       "7657063     7657063          2064 -11.209008  12.041980    NaN   \n",
       "7657064     7657064          2065 -11.125675  12.041980    NaN   \n",
       "\n",
       "                                           geometry  index__spam  ID_spam  \\\n",
       "0        POINT (-15.9900569732419 16.6937073347074)          NaN      NaN   \n",
       "1        POINT (-15.9897874786566 16.6937073347074)          NaN      NaN   \n",
       "2        POINT (-15.9895179840714 16.6937073347074)          NaN      NaN   \n",
       "3        POINT (-15.9253782727816 16.6937073347074)          NaN      NaN   \n",
       "4        POINT (-15.9251087781964 16.6937073347074)          NaN      NaN   \n",
       "...                                             ...          ...      ...   \n",
       "7657060                 POINT (-11.459007 12.04198)          NaN   2507.0   \n",
       "7657061                 POINT (-11.375674 12.04198)          NaN   2508.0   \n",
       "7657062                 POINT (-11.292341 12.04198)          NaN   2509.0   \n",
       "7657063                 POINT (-11.209008 12.04198)          NaN   2510.0   \n",
       "7657064                 POINT (-11.125675 12.04198)          NaN   2511.0   \n",
       "\n",
       "              val                                           geom_UTM  \\\n",
       "0             0.0                                                NaN   \n",
       "1             0.0                                                NaN   \n",
       "2             0.0                                                NaN   \n",
       "3             0.0                                                NaN   \n",
       "4             0.0                                                NaN   \n",
       "...           ...                                                ...   \n",
       "7657060  411859.0  MULTIPOLYGON (((890065.2102591199 1329046.5648...   \n",
       "7657061  497200.0  MULTIPOLYGON (((899153.5281643758 1329166.2613...   \n",
       "7657062  217938.0  MULTIPOLYGON (((908242.6248140114 1329288.7282...   \n",
       "7657063  201222.0  MULTIPOLYGON (((917332.5179180377 1329413.9667...   \n",
       "7657064  944829.0  MULTIPOLYGON (((917332.5179180377 1329413.9667...   \n",
       "\n",
       "                                             geom_WGS84  lc_sum  spam_V  \\\n",
       "0                                                   NaN     NaN     0.0   \n",
       "1                                                   NaN     NaN     0.0   \n",
       "2                                                   NaN     NaN     0.0   \n",
       "3                                                   NaN     NaN     0.0   \n",
       "4                                                   NaN     NaN     0.0   \n",
       "...                                                 ...     ...     ...   \n",
       "7657060   POINT (-11.45900694020285 12.041979526954448)     0.0     inf   \n",
       "7657061  POINT (-11.375673926651558 12.041979526573865)     0.0     inf   \n",
       "7657062  POINT (-11.292340913098172 12.041979526183944)     0.0     inf   \n",
       "7657063  POINT (-11.209007899992315 12.041979525784605)     0.0     inf   \n",
       "7657064   POINT (-11.125674886884132 12.04197952537604)     0.0     inf   \n",
       "\n",
       "                 NN       NN_dist  \n",
       "0        5343951929  14607.791436  \n",
       "1        5343951929  14628.808767  \n",
       "2        5343951929  14649.852284  \n",
       "3        3639403839  14029.850649  \n",
       "4        3639403839  14018.013147  \n",
       "...             ...           ...  \n",
       "7657060  2241013982  14274.586682  \n",
       "7657061  6662833878  13597.325940  \n",
       "7657062  6662833878  14994.531096  \n",
       "7657063  2217989971  20260.745415  \n",
       "7657064  3344553248  24408.027959  \n",
       "\n",
       "[7657065 rows x 15 columns]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ag_snap = os.path.join(out_pth, \"ag_snap.csv\")\n",
    "ag_snap = pd.read_csv(ag_snap)\n",
    "ag_snap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13941124 entries, 0 to 13941123\n",
      "Data columns (total 12 columns):\n",
      " #   Column        Dtype   \n",
      "---  ------        -----   \n",
      " 0   Unnamed: 0    int64   \n",
      " 1   index         int64   \n",
      " 2   ID_spam       float64 \n",
      " 3   grid_val      float64 \n",
      " 4   ID_LC         float64 \n",
      " 5   val           float64 \n",
      " 6   x             float64 \n",
      " 7   y             float64 \n",
      " 8   Unnamed: 0.1  float64 \n",
      " 9   geometry      geometry\n",
      " 10  NN            int64   \n",
      " 11  NN_dist       float64 \n",
      "dtypes: float64(8), geometry(1), int64(3)\n",
      "memory usage: 1.2 GB\n"
     ]
    }
   ],
   "source": [
    "ag_snap.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>index</th>\n",
       "      <th>ID_spam</th>\n",
       "      <th>grid_val</th>\n",
       "      <th>ID_LC</th>\n",
       "      <th>val</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>Unnamed: 0.1</th>\n",
       "      <th>geometry</th>\n",
       "      <th>NN</th>\n",
       "      <th>NN_dist</th>\n",
       "      <th>HDurban1</th>\n",
       "      <th>HDurban2</th>\n",
       "      <th>HDurban3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787187.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.562504</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.56250 16.67094)</td>\n",
       "      <td>3656618126</td>\n",
       "      <td>5113.670395</td>\n",
       "      <td>163.502985</td>\n",
       "      <td>164.457268</td>\n",
       "      <td>170.140815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13818608.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.554419</td>\n",
       "      <td>16.682254</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.55442 16.68225)</td>\n",
       "      <td>3656617769</td>\n",
       "      <td>5632.260552</td>\n",
       "      <td>163.910812</td>\n",
       "      <td>164.865095</td>\n",
       "      <td>170.548642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787063.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.559809</td>\n",
       "      <td>16.676864</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.55981 16.67686)</td>\n",
       "      <td>3656617751</td>\n",
       "      <td>5453.608825</td>\n",
       "      <td>163.568214</td>\n",
       "      <td>164.522497</td>\n",
       "      <td>170.206044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787188.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.561965</td>\n",
       "      <td>16.670935</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.56197 16.67094)</td>\n",
       "      <td>3656618126</td>\n",
       "      <td>5075.881917</td>\n",
       "      <td>163.502985</td>\n",
       "      <td>164.457268</td>\n",
       "      <td>170.140815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2487.0</td>\n",
       "      <td>13787225.0</td>\n",
       "      <td>4.236797e+00</td>\n",
       "      <td>-15.571128</td>\n",
       "      <td>16.668240</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POINT (-15.57113 16.66824)</td>\n",
       "      <td>3656618126</td>\n",
       "      <td>5568.582130</td>\n",
       "      <td>163.502985</td>\n",
       "      <td>164.457268</td>\n",
       "      <td>170.140815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941119</th>\n",
       "      <td>13941119</td>\n",
       "      <td>118</td>\n",
       "      <td>5.0</td>\n",
       "      <td>54822.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>5.482200e+04</td>\n",
       "      <td>-15.375659</td>\n",
       "      <td>16.791961</td>\n",
       "      <td>118.0</td>\n",
       "      <td>POINT (-15.37566 16.79196)</td>\n",
       "      <td>3651042393</td>\n",
       "      <td>4280.425149</td>\n",
       "      <td>193.975805</td>\n",
       "      <td>194.930088</td>\n",
       "      <td>200.613635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941120</th>\n",
       "      <td>13941120</td>\n",
       "      <td>119</td>\n",
       "      <td>2383.0</td>\n",
       "      <td>290657.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>2.906570e+05</td>\n",
       "      <td>-12.873696</td>\n",
       "      <td>12.212042</td>\n",
       "      <td>119.0</td>\n",
       "      <td>POINT (-12.87370 12.21204)</td>\n",
       "      <td>4818951142</td>\n",
       "      <td>9019.155622</td>\n",
       "      <td>219.410253</td>\n",
       "      <td>220.599554</td>\n",
       "      <td>224.281252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941121</th>\n",
       "      <td>13941121</td>\n",
       "      <td>120</td>\n",
       "      <td>2382.0</td>\n",
       "      <td>743827.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>7.438270e+05</td>\n",
       "      <td>-12.955614</td>\n",
       "      <td>12.224018</td>\n",
       "      <td>120.0</td>\n",
       "      <td>POINT (-12.95561 12.22402)</td>\n",
       "      <td>4618077921</td>\n",
       "      <td>10786.551522</td>\n",
       "      <td>199.346845</td>\n",
       "      <td>200.536147</td>\n",
       "      <td>204.217845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941122</th>\n",
       "      <td>13941122</td>\n",
       "      <td>121</td>\n",
       "      <td>2345.0</td>\n",
       "      <td>94605.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>9.460500e+04</td>\n",
       "      <td>-16.375655</td>\n",
       "      <td>12.208646</td>\n",
       "      <td>121.0</td>\n",
       "      <td>POINT (-16.37565 12.20865)</td>\n",
       "      <td>3507831510</td>\n",
       "      <td>2881.578459</td>\n",
       "      <td>88.413319</td>\n",
       "      <td>90.114132</td>\n",
       "      <td>215.944545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13941123</th>\n",
       "      <td>13941123</td>\n",
       "      <td>122</td>\n",
       "      <td>2443.0</td>\n",
       "      <td>1451706.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.451706e+06</td>\n",
       "      <td>-12.375670</td>\n",
       "      <td>12.125313</td>\n",
       "      <td>122.0</td>\n",
       "      <td>POINT (-12.37567 12.12531)</td>\n",
       "      <td>3882487056</td>\n",
       "      <td>398.111940</td>\n",
       "      <td>237.810003</td>\n",
       "      <td>238.999304</td>\n",
       "      <td>243.038501</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13941124 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          Unnamed: 0  index  ID_spam   grid_val       ID_LC           val  \\\n",
       "0                  0      0      9.0     2487.0  13787187.0  4.236797e+00   \n",
       "1                  1      1      9.0     2487.0  13818608.0  4.236797e+00   \n",
       "2                  2      2      9.0     2487.0  13787063.0  4.236797e+00   \n",
       "3                  3      3      9.0     2487.0  13787188.0  4.236797e+00   \n",
       "4                  4      4      9.0     2487.0  13787225.0  4.236797e+00   \n",
       "...              ...    ...      ...        ...         ...           ...   \n",
       "13941119    13941119    118      5.0    54822.0         0.1  5.482200e+04   \n",
       "13941120    13941120    119   2383.0   290657.0         0.1  2.906570e+05   \n",
       "13941121    13941121    120   2382.0   743827.0         0.1  7.438270e+05   \n",
       "13941122    13941122    121   2345.0    94605.0         0.1  9.460500e+04   \n",
       "13941123    13941123    122   2443.0  1451706.0         0.1  1.451706e+06   \n",
       "\n",
       "                  x          y  Unnamed: 0.1                    geometry  \\\n",
       "0        -15.562504  16.670935           NaN  POINT (-15.56250 16.67094)   \n",
       "1        -15.554419  16.682254           NaN  POINT (-15.55442 16.68225)   \n",
       "2        -15.559809  16.676864           NaN  POINT (-15.55981 16.67686)   \n",
       "3        -15.561965  16.670935           NaN  POINT (-15.56197 16.67094)   \n",
       "4        -15.571128  16.668240           NaN  POINT (-15.57113 16.66824)   \n",
       "...             ...        ...           ...                         ...   \n",
       "13941119 -15.375659  16.791961         118.0  POINT (-15.37566 16.79196)   \n",
       "13941120 -12.873696  12.212042         119.0  POINT (-12.87370 12.21204)   \n",
       "13941121 -12.955614  12.224018         120.0  POINT (-12.95561 12.22402)   \n",
       "13941122 -16.375655  12.208646         121.0  POINT (-16.37565 12.20865)   \n",
       "13941123 -12.375670  12.125313         122.0  POINT (-12.37567 12.12531)   \n",
       "\n",
       "                  NN       NN_dist    HDurban1    HDurban2    HDurban3  \n",
       "0         3656618126   5113.670395  163.502985  164.457268  170.140815  \n",
       "1         3656617769   5632.260552  163.910812  164.865095  170.548642  \n",
       "2         3656617751   5453.608825  163.568214  164.522497  170.206044  \n",
       "3         3656618126   5075.881917  163.502985  164.457268  170.140815  \n",
       "4         3656618126   5568.582130  163.502985  164.457268  170.140815  \n",
       "...              ...           ...         ...         ...         ...  \n",
       "13941119  3651042393   4280.425149  193.975805  194.930088  200.613635  \n",
       "13941120  4818951142   9019.155622  219.410253  220.599554  224.281252  \n",
       "13941121  4618077921  10786.551522  199.346845  200.536147  204.217845  \n",
       "13941122  3507831510   2881.578459   88.413319   90.114132  215.944545  \n",
       "13941123  3882487056    398.111940  237.810003  238.999304  243.038501  \n",
       "\n",
       "[13941124 rows x 15 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ag_to_HDurban = pd.merge(ag_snap, HDurban_all, on='NN',how='left')\n",
    "ag_to_HDurban"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 13941124 entries, 0 to 13941123\n",
      "Data columns (total 15 columns):\n",
      " #   Column        Dtype   \n",
      "---  ------        -----   \n",
      " 0   Unnamed: 0    int64   \n",
      " 1   index         int64   \n",
      " 2   ID_spam       float64 \n",
      " 3   grid_val      float64 \n",
      " 4   ID_LC         float64 \n",
      " 5   val           float64 \n",
      " 6   x             float64 \n",
      " 7   y             float64 \n",
      " 8   Unnamed: 0.1  float64 \n",
      " 9   geometry      geometry\n",
      " 10  NN            int64   \n",
      " 11  NN_dist       float64 \n",
      " 12  HDurban1      float64 \n",
      " 13  HDurban2      float64 \n",
      " 14  HDurban3      float64 \n",
      "dtypes: float64(11), geometry(1), int64(3)\n",
      "memory usage: 1.7 GB\n"
     ]
    }
   ],
   "source": [
    "ag_to_HDurban.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 13941124 entries, 0 to 13941123\n",
      "Data columns (total 11 columns):\n",
      " #   Column    Dtype   \n",
      "---  ------    -----   \n",
      " 0   ID_ag     int64   \n",
      " 1   ID_spam   float64 \n",
      " 2   grid_val  float64 \n",
      " 3   ID_LC     float64 \n",
      " 4   val       float64 \n",
      " 5   x         float64 \n",
      " 6   y         float64 \n",
      " 7   geometry  geometry\n",
      " 8   NN        int64   \n",
      " 9   NN_dist   float64 \n",
      " 10  ag_HDurb  float64 \n",
      "dtypes: float64(8), geometry(1), int64(2)\n",
      "memory usage: 1.2 GB\n"
     ]
    }
   ],
   "source": [
    "ag_to_HDurban = ag_to_HDurban.drop(columns=['index','Unnamed: 0.1', 'HDurban2', 'HDurban3'])\n",
    "ag_to_HDurban.rename(columns={'Unnamed: 0': 'ID_ag', 'HDurban1':'ag_HDurb'}, inplace=True)\n",
    "ag_to_HDurban.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "ag_to_HDurban.to_csv(os.path.join(out_pth, 'fromagriculture/ag_to_HDurban.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End of script. Load into QGIS or Arc and visualize at 10 min intervals. \n",
    "QML file for symbology in QGIS:\n",
    "R:\\GEOGlobal\\Design\\symb_traveltimes_10min.qml"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}