Chapter 07 - Cross-Validation in Finance.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as mpl\n",
    "\n",
    "from collections import defaultdict\n",
    "from functools import reduce\n",
    "from path import Path\n",
    "from pprint import pprint\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline\n",
    "mpl.style.use('ggplot')\n",
    "mpl.rcParams['figure.figsize'] = 16,6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.1\n",
    "\n",
    "Why is shuffling a dataset before conducting a k-fold CV generall a bad idea in finance? What is the purpose of shuffling? Why does shuffling defeat the purpose of k-fold CV in financial datasets?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Markets are adaptive systems and thus sample order is meaningful. Generally we would shuffle data because we don't want the model to make predictions based on the order of the data.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sampling import dollar_bars\n",
    "from filters import cusum\n",
    "from multiprocess import mpPandasObj\n",
    "from load_data import load_contracts\n",
    "from labeling import getEvents, getVerticalBarriers, getBins\n",
    "from util import getDailyVol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "es_contracts = load_contracts('@ES')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.2a\n",
    "\n",
    "Take a pair of matrices (X, y) representing observed features and labels. These could be one of the datasets derived from the exercises in Chapter 3.\n",
    "\n",
    "Derive the performance from a 10-fold CV of an RF classifier on (X, y), without shuffling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\doda\\Dropbox\\algotrading\\AFML\\labeling.py:7: FutureWarning: \n",
      "Passing list-likes to .loc or [] with any missing label will raise\n",
      "KeyError in the future, you can use .reindex() as an alternative.\n",
      "\n",
      "See the documentation here:\n",
      "http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike\n",
      "  trgt = trgt.loc[tEvents]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=5, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=512, n_jobs=None,\n",
       "            oob_score=True, random_state=42, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dbars = dollar_bars(es_contracts, 100000000)\n",
    "\n",
    "df = dbars.copy()['2018-8-1':'2019-5-1']\n",
    "close = df['Close']\n",
    "daily_vol = getDailyVol(close)\n",
    "tEvents = cusum(close, getDailyVol(close).mean())\n",
    "t1 = getVerticalBarriers(close, tEvents, numDays=1)\n",
    "events = getEvents(close, tEvents=tEvents, ptSl=[1,1], t1=t1, trgt=daily_vol, minRet=0.01)\n",
    "\n",
    "close = df['Close']\n",
    "fast_ma = close.rolling(50).mean()\n",
    "slow_ma = close.rolling(200).mean()\n",
    "\n",
    "long_signals = (fast_ma >= slow_ma)\n",
    "short_signals = (fast_ma < slow_ma)\n",
    "\n",
    "df.loc[long_signals, 'side'] = 1\n",
    "df.loc[short_signals, 'side'] = -1\n",
    "events['side'] = df['side']\n",
    "\n",
    "bins = getBins(events, df['Close'])\n",
    "bins['bin'].value_counts()\n",
    "\n",
    "df['log_ret'] = np.log(close).diff()\n",
    "df['vol5'] = df['log_ret'].rolling(5).std()\n",
    "df['vol10'] = df['log_ret'].rolling(10).std()\n",
    "\n",
    "df['serialcorr20-1'] = df['log_ret'].rolling(20).apply(lambda x: pd.Series(x).autocorr(lag=1))\n",
    "\n",
    "\n",
    "df = df.shift()\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "train_features = df.loc[events.index][['side', 'vol5', 'serialcorr20-1']]\n",
    "train_labels = bins['bin']\n",
    "\n",
    "rf = RandomForestClassifier(n_estimators=512, random_state=42, max_depth=5, oob_score=True)\n",
    "rf.fit(train_features, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate, KFold\n",
    "\n",
    "cvd = cross_validate(rf, train_features, train_labels, cv=KFold(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6983229813664596"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvd['test_score'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.2b\n",
    "\n",
    "Take a pair of matrices (X, y) representing observed features and labels. These could be one of the datasets derived from the exercises in Chapter 3.\n",
    "\n",
    "Derive the performance from a 10-fold CV of an RF classifier on (X, y), with shuffling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cvd = cross_validate(rf, train_features, train_labels, cv=KFold(10, shuffle=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7011180124223603"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvd['test_score'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.2c\n",
    "\n",
    "Take a pair of matrices (X, y) representing observed features and labels. These could be one of the datasets derived from the exercises in Chapter 3.\n",
    "\n",
    "Why are both results so different?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Shuffling introduces information from the future into our training set.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.2d\n",
    "\n",
    "Take a pair of matrices (X, y) representing observed features and labels. These could be one of the datasets derived from the exercises in Chapter 3.\n",
    "\n",
    "How does shuffling leak information?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: The model is able to train on data from the future.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.3a\n",
    "\n",
    "Take the same pair of matrices (X, y) you used in exercise 2.\n",
    "\n",
    "Derive the performance from a 10-fold purged CV of an RF on (X, y), with 1% embargo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The mean CV performance is 0.70.\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection._split import _BaseKFold\n",
    "from sklearn.metrics import log_loss, accuracy_score\n",
    "from cv import cvScore\n",
    "\n",
    "cvd = cvScore(rf, train_features, train_labels, pctEmbargo=0.01,\n",
    "              sample_weight=None, cv=10,\n",
    "              t1=pd.Series(train_features.index, index=train_features.index),\n",
    "              scoring='accuracy'\n",
    "             )\n",
    "print(\"The mean CV performance is {:.2f}.\".format(np.mean(cvd)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6UAAAFpCAYAAABklI6gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3XeYVNXh//H3ubvswrKAwIIIigqCiBVBQVRERcASW2wkGo1RoyYxJvb0X9qXqCTGxBJ7osYSY4kdsWFXBCuKvSAgXTrC3vP7YwyJsdB298zsvl/Psw87zGX2s851Zz57zj0nxBgjkiRJkiQlkKUOIEmSJElquiylkiRJkqRkLKWSJEmSpGQspZIkSZKkZCylkiRJkqRkLKWSJEmSpGQspZIkSZKkZCylkiRJkqRkLKWSJEmSpGQspZIkSZKkZCylkiRJkqRkylN+8SlTpqT88qonNTU1zJw5M3UMNRKeT6pLnk+qS55PqkueT6pLxXI+de7ceZWOc6RUkiRJkpSMpVSSJEmSlIylVJIkSZKUjKVUkiRJkpSMpVSSJEmSlIylVJIkSZKUjKVUkiRJkpSMpVSSJEmSlIylVJIkSZKUjKVUkiRJkpSMpVSSJEmSlIylVJIkSZKUjKVUkiRJkpRMeeoAkiQ1FfnYe1JHUB1aVF1NvmBB6hgrlQ0anjqCJH0pR0olSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyVhKJUmSJEnJWEolSZIkSclYSiVJkiRJyZSv7ICZM2dywQUXMHfuXEIIDBkyhL322utTx7z88sucffbZdOzYEYD+/ftz0EEH1U9iSZIkSVKjsdJSWlZWxhFHHEG3bt1YvHgxZ555JltttRXrr7/+p47bbLPNOPPMM+stqCRJkiSp8Vnp9N22bdvSrVs3AFq0aEGXLl2YPXt2vQeTJEmSJDV+Kx0p/W/Tp0/n7bffZpNNNvnMfa+99hqnnXYabdu25YgjjmCDDTb4zDFjxoxhzJgxAIwcOZKampo1jK1iVl5e7nOrOuP5pLqU+nxaVF2d7Gur7pVlZVSXwHNa5c/QkpD655Mal1I7n0KMMa7KgUuWLOHnP/85Bx54IP379//UfYsWLSLLMpo3b8748eO56qqrOP/881f6mFOmTFmz1CpqNTU1zJw5M3UMNRKeT6pLqc+nfOw9yb626l51dTULFixIHWOlskHDU0fQKkj980mNS7GcT507d16l41Zp9d3ly5czatQodt55588UUoCqqiqaN28OwLbbbkttbS3z5s1bjbiSJEmSpKZopaU0xsjFF19Mly5d2GeffT73mLlz5/LvAdc33niDPM9p1apV3SaVJEmSJDU6K72mdNKkSYwdO5auXbty2mmnATBixIgVw8FDhw7lySefZPTo0ZSVlVFRUcHJJ59MCKF+k0uSJEmSSt5KS2mvXr248cYbv/SY4cOHM3y41ytIkiRJklbPKl1TKkmSJElSfbCUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpKxlEqSJEmSkrGUSpIkSZKSsZRKkiRJkpIpTx1AkiSp1MUY4eOlsGghLF4EixfC8mVQUfnJR3OorFxxO2SOC0jSv1lKJUmSVkPMc5g1g6WTXiS+9zbM/6hQRPPaVXuAEIht20PHzrDuetCxM6F5i/oNLUlFzFIqSZL0JWKMMHc2TJsM0z6AD6fAso/5GKBte+jYCVq0LHxUVf3n82bNCqOn//5YuqTw5+JFMPNDeH0ivPpC4Wu0aQsd14MuG0GXro6kSmpSLKWSJEmfIy5bBu+8Dq+9BLNnFv6yVWvYaBPotD4tu/VgUR6//EFaVH3x49fWwuwZhZI7fQq880ahqLasJvbYHHr0dgRVUpNgKZUkSfovce7sQhF96zVY9nFhNHT7naHLRoTqViuOy6pawoIFa/x1QlkZdOhU+GDbwrTgyW/DpJfguafghWeIG20CPbeEmo6EEOrgu5Ok4mMplSRJTV6MESa/AxOfg+lTIctgw02g5+bQoVODFMKQZdC1O3Tt/l/FeFKhHLfvSNx2B0KnLvWeQ5IamqVUkiQ1aXH6VBj/BMyYBtWtYdsdoHuvpFNnwzrtYPtBxD4DCqX05Qlw323EDTeBvgMJLauTZZOkumYplSRJTVL8aA5MeBLef7tw7eeAwYUyWkSLDIVmFbDpFsTumxaK6csTYPI7xC37Qu9tClOAJanEWUolSVKTEhcvguefhjdegbJy2Hp72GxrQrNmqaN9oVDeDLbentitFzz7WOGa0zdfJW63E6HLhqnjSdJasZRKkqQmIcYIr71cmKpbW1u4XnTLfoQvWSG32IRWrWHwnsQp78Ezj8IDdxI37gH9Bxd1qZakL2MplSRJjV6cPw+eeBA+/AA6rQ/9BxFar5M61hoLnbsS9zkUXh4PL4yD2TOJg4YVrkWVpBJjKZUkSY1WYXT0pcLoaAiF60Y32axRbK8Syspgq+2IHTvDI6PhrpuIA3YhdNs0dTRJWi2WUkmS1CjF+R99Mjo6BdbbAHYYTGjZauX/sMSETl2Iex8Cj94Hj91P/HAqbL8Tocy3eZJKw0p/Ws2cOZMLLriAuXPnEkJgyJAh7LXXXp86JsbIlVdeyYQJE6isrOTEE0+kW7du9RZakiTpi8QYC4sYPfNoYb/RHXYtrKrbCEZHv0ioakkcsm9hAaeXxsOs6YXpvK3bpI4mSSu10lJaVlbGEUccQbdu3Vi8eDFnnnkmW221Feuvv/6KYyZMmMC0adM4//zzef3117nsssv47W9/W6/BJUmS/ldctgyeHgtvTSpcOzpwtyazp2fIMugzgNihEzx2P9x9E3HXvVb+DyUpsZVuxNW2bdsVo54tWrSgS5cuzJ49+1PHjBs3jkGDBhFCoGfPnixcuJA5c+bUT2JJkqTPET+aA3f/s1BIt9oOdt+nyRTS/xbW3wj2Ohgqm8OY24kvjksdSZK+1GrtDj19+nTefvttNtlkk0/9/ezZs6mpqVlxu3379p8prpIkSfUlvvM63HUTLFkEu3+FsPV2hZHDJiq0ag3DDoA265Bf8Bvypx5OHUmSvtAqXwG/ZMkSRo0axVFHHUVV1af384oxfub4z7tuY8yYMYwZMwaAkSNHfqrIqvEoLy/3uVWd8XxSXUp9Pi2qbnqjdvUt1i5n6eMPseyl8WTrdqbF0P3IqhtmMaOyrIzqYn5Oq6uJB3ydpU89wrLLRlFFTtXeB6dOpS+Q+ueTGpdSO59WqZQuX76cUaNGsfPOO9O/f//P3N++fXtmzpy54vasWbNo27btZ44bMmQIQ4YMWXH7v/+NGo+amhqfW9UZzyfVpdTnU75gQbKv3RjFxYvgobth5oew2dbk2w5gEQEa6L9zdXU1C0rgOQ0n/gguOZf5l/2BBdOmEvYd0agXfSpVqX8+qXEplvOpc+fOq3TcSue1xBi5+OKL6dKlC/vss8/nHtOvXz/Gjh1LjJHXXnuNqqqqzy2lkiRJdSHOmVmYrjtnFgwaRui3IyErSx2rKIVmFWTHn0HYcQjxjuuJf/8LMa9NHUuSVljpSOmkSZMYO3YsXbt25bTTTgNgxIgRK5r30KFD6dOnD+PHj+ekk06ioqKCE088sX5TS5KkJitOfgceuQ+aVcCwAwjtO6SOVPRCWRkc+T2obk2892bIczj8BEdMJRWFlZbSXr16ceONN37pMSEEjjnmmDoLJUmS9L9ijPDqC/Ds49C2Bnbdk1BVxNd0FpkQAuGgo8izQLz7n9CyJeHAI1PHkqRVX+hIkiQplZjXwtOPwOsToWs3GLg7oVmz1LFKUjjgG7BoIfHuf5K3qCbb86upI0lq4iylkiSpqMWPl8LD98K0ybB5H+gzwGmnayGEAF/7NixeRLz5r+RVLcl2GZ46lqQmzFIqSZKKVly0EB64Az6aAwN3I3TvlTpSoxCyMvjmycTFi4jXXkTeoops+0GpY0lqoprurtKSJKmoxY/mwD03w/x5sOveFtI6FsrLyY4/A3r0Jl7xB+KL41JHktREWUolSVLRiTM/hHtvgeXLYeh+hM4bpI7UKIWKSrLv/hS6bER+0Ujiay+ljiSpCbKUSpKkohI/eA9G31bY8mX4AYT2HVNHatRCiyqyk38B7TuSX/Bb4vQpqSNJamIspZIkqWjEtybBg3dB63Vg+IGE1uukjtQkhFZtyE76GWSB/E+/LlzLK0kNxFIqSZKKQpz4HDx2P6y7Hgzdn9CiKnWkJiV06ER2/FkwYyr5pecQa2tTR5LURFhKJUlSUjFG4vPPwLOPQ9fusNs+hIqK1LGapLDpFoSvHQ8vjSfedFXqOJKaCLeEkSRJycQYYfzjMPF56N4LBgwmZP7OPKVs0DDyKe8Rx9xG3nkDsp2Hpo4kqZGzlEpqcvKx96SOoEQWVVeTL1iQOoY+EWOEp8fCay/DplvCdjsRQkgdS0A4+GjitMnEay8mrtuZ0HOL1JEkNWL+KlKSJDW4mOfw+P2FQrp5HwtpkQllZWTHnQYd1i1sFTNjWupIkhoxS6kkSWpQsbYWHhkNb70G2/QnbLuDhbQIharqwh6meU7+518TlyxOHUlSI2UplSRJDSYuXwYP3Q3vvQX9diRs2Td1JH2JsG5nsm+fDlMnE6+9qDDlWpLqmKVUkiQ1iLjsY3jgTpjyXmFBo822Th1JqyD03obwlcOITz5EfPS+1HEkNUKWUkmSVO/i0iVw379g+jTYaQ9Cj96pI2k1hL0Pht7bEK+7hDj57dRxJDUyllJJklSv4uJFcN9tMGcm7DKMsHGP1JG0mkJWRvatH0LLavKLzyYuWZQ6kqRGxFIqSZLqTVy4AEbfCvM+gt32JmywcepIWkOh9Tpkx54K06cS/3aB15dKqjOWUkmSVC/i/I/g3ltg8SIY8hXCehukjqS1FHpuQdj/68RnHiE+7J7PkuqGpVSSJNW5+NFsuPdWWL4M9tiX0HG91JFUR8Lwr8IWfYk3XEp8983UcSQ1ApZSSZJUp+LsGYVCSoQ99iO075g6kupQyDKyo38ArdYh/8vviIsWpo4kqcRZSiVJUp2J06fC6NugvByGHkBo2z51JNWD0Ko12XGnwqzpxGsvSh1HUomzlEqSpDoRp74PY26HFlUw7ABC6zapI6kehU16E74ygvj0WPKnHk4dR1IJs5RKkqS1Ft9/Gx64E1q1hqH7E1q2Sh1JDSDseRB070W89mLirBmp40gqUZZSSZK0VuLbr8PD90DbmkIhbVGVOpIaSCgrK1xfmufkV55HzPPUkSSVIEupJElaY/H1ifDofdBxvcIqu5XNU0dSAwsd1yMcdgxMepF4322p40gqQZZSSZK0RuLE5+HJh6BzV9htH0KzitSRlEjYcQj0GUC89erCVG5JWg2WUkmStFpijMQXnoFnH4Ou3WHwnoTy8tSxlFAIgeyI70LLVuSXjSIu+zh1JEklxFIqSZJWWYwRxj8Bzz8D3XvBznsQyspSx1IRCK1akx11Ekx5j3jz1anjSCohllJJkrRKYozw1MMw8TnYdEvYYVdC5lsJ/UfYoi9h172IY24jTnwudRxJJcJXEkmStFIxz+GxMfD6RNhiW9huJ0IIqWOpCIWvfhM6rU9+5R+JixamjiOpBFhKJUnSl4q1y2HsvfD267BNf0KfARZSfaFQWVnYJuajOcSbrkwdR1IJsJRKkqQvFD9eCvffAe+/XRgd3bJv6kgqAWHjHoRhBxAfGU18eULqOJKKnKVUkiR9rrhoIYy+FWZMg52GEHptlTqSSkjYd0RhGu/f/kxcvCh1HElFzFIqSZI+I340B+65GebPg133JmzcM3UklZjQrKKwGu+cmcSbrkodR1IRs5RKkqRPiTOmwb23QO1yGLo/ofMGqSOpRIXuvQh77Eccew/xledTx5FUpCylkiRphTj5HbjvX1BRAcMPJLTvkDqSSlzY7+vQsTP5X/9EXLI4dRxJRchSKkmSAIivT4SH7oY2bWHYgYRWbVJHUiMQKioL03hnzyDe/NfUcSQVIUupJElNXIyROP4JePIhWG99GLofoUVV6lhqREKP3oTd9iE+eBdx0kup40gqMpZSSZKasLhsGTx8D7w8AXpuXljUqFlF6lhqhMIBR0CHTuR/PZ+4dEnqOJKKiKVUkqQmasWWL5PfgX47wvaDCJlvDVQ/QmVzsiNPghnTiP+6LnUcSUXEVx5JkpqgOHsG3H0TzJsLg/ckbLY1IYTUsdTIhU23IAwaRrzvNuK7b6aOI6lIWEolSWpi4vtvF7Z8IcDwAwjrb5Q6kpqQ8NUjoXUb8r/9iVhbmzqOpCJgKZUkqYmIMRKff/qTFXbbwZ5fJbStSR1LTUyoqiYbcRy89xbx/n+ljiOpCFhKJUlqAuLSJfDgnfDCOOi2aWGF3aqWqWOpqdp2IGy9PfG2vxNnTEudRlJillJJkhq5OHsm3HUTTJ0M2w+CgbsRypuljqUmLIRA9rVvQ8jIr72IGGPqSJISKl/ZARdeeCHjx4+nTZs2jBo16jP3v/zyy5x99tl07NgRgP79+3PQQQfVfVJJkrTa4puvwlMPQ2VzGLo/oUOn1JEkAEK7DoQDjyBedwnxqYcJAwanjiQpkZWW0sGDBzN8+HAuuOCCLzxms80248wzz6zTYJIkac3F2loY9xi89hKs2xl2HkpoUZU6lvQpYfCexCcfIt5wGXHzbQmtWqeOJCmBlU7f7d27N9XV1Q2RRZIk1YH40Wy4+5+FQtp7Gxiyr4VURSlkZWTf+C4sXkj8xxWp40hKpE6uKX3ttdc47bTT+O1vf8v7779fFw8pSZJWU4yROOkluPMfsGhBYf/RvgMJmUtIqHiF9TciDDuQ+MQDxInPpY4jKYGVTt9dmY033pgLL7yQ5s2bM378eM455xzOP//8zz12zJgxjBkzBoCRI0dSU+My9I1ReXm5z63qTH2cT4uc/dFklWVljXb2T75oIUseuofad9+ibIONab7bnmRVjfN7LRalcj5VlcBrcjzyRGZNeBKuv4T2511NqKhMHanB+f5JdanUzqe1LqVVVf+ZDrTtttty+eWXM2/ePFq3/uw1AUOGDGHIkCErbs+cOXNtv7yKUE1Njc+t6kx9nE/5ggV1+ngqHdXV1SxohM9/nPwOPPEgfPwx9NuJ2l5bsigHGuH3WkxK5XxaVCKvyfGwY8n/8DNmXHMJ2b4jUsdpcL5/Ul0qlvOpc+fOq3TcWs/nmTt37oplvN944w3yPKdVq1Zr+7CSJGkl4sdLiU8+BA/eBS2qYK+DCJttRQghdTRptYXe2xC225l4903E6VNSx5HUgFY6UnreeecxceJE5s+fz/HHH88hhxzC8uXLARg6dChPPvkko0ePpqysjIqKCk4++WRfDCVJqkcxRnjvTXjmUViyGDbbGvr0J5St9QQoKalwyLeILz1Lfu1fyE7+he8ppSZipa9eJ5988pfeP3z4cIYPH15ngSRJ0heLC+bB02Phg/egXQfYdS9C+46pY0l1IqzTjrDf4cTrLyGOe4yw3U6pI0lqAP5KVZKkEhDzHF55Hp5/BgLQd0fotaUr66rRCbvuSXz8/sLepVts63ZGUhPgK5kkSUUuTp0Md/0Dxj8B660P+44g9N7aQqpGKWRlZIefAPPmEG+7NnUcSQ3AkVJJkopUnD0TJjwBU96HqmrYZRhs0M3r7NTohY17EgYNIz5wJ3Hg7oSu3VJHklSP/BWrJElFJi6YR3x0DNx5I8ycDn0Hwv5fI3TtbiFVkxEO+AZUtyK/9qLC9HVJjZYjpZIkFYm4ZDG8NB4mvQghwOZ9YIttCRWVqaNJDS60rCYcfDTxij8QHx1NGOTCmlJjZSmVJCmxOG9uYRGjN1+FPIdum8LW2xNaVqeOJiUVBgwmPjaG+M+/EbcZQGi9TupIkuqB03clSUokTp9KfOhuuO3v8MYrsHFP+MphhIG7WUglIIRA9vXjYekS4k1XpY4jqZ44UipJUgOKtbUw+R145TmY8SFUVMKWfWHTLd36QvocYb0NCEP3J959E3GnIYSeW6SOJKmOWUolSapnMcbCgkVvTYJ3XoePl0J1a9huZ+jei9CsWeqIUlELex9KfHos+TUXkf3sPEK5/89IjYmlVJKkehIXzIe3J8Fbr8G8uVBWDhtsDN03hU7ru8+otIpCZSXZiG+T//lXxPv+Rdjzq6kjSapDllJJkupIzHOY+SF88B5MeQ9mzyjcsW7nwkq6XbsTKirShpRKVNh6O9hmAPGO64nb70xo3zF1JEl1xFIqSdIaijHCooUwbXKhiE59vzA1NwTo0Am26Q8b9yBUt04dVWoUssOOJf/ZieTXX0rZd36cOo6kOmIplSRpFcWlS2DWDJj1YeEa0VnTYfGiwp0tqgpTc7t0LUzNrWyeNqzUCIX2HQj7jiDedBXx+acJW2+fOpKkOmAplSTpv8S8FhYsgPkfwfy5hT/n/fvzef85sPU60Gl9qOkIHdeDtjWEENIFl5qIsPu+xMcfIL/uErJeW/kLIKkRsJRKklaIeV4Y+VuyGJYuLvz574+lS2DZMqhdDsuXQ+2ywp/Ll0OeQ4xA/OSB+M/nWQZZWeGjrAzK/ud2VlY4puy/bn/p5+Wf/vuyMggZfF4fjBFqaz/Ju5xlzcqJCxcWbn+8BJYs+a/vb/Entxd98r18orwZtGoDbTtA980KJbR9R0JFZf0+GZI+VygvJ/v6CeTnnEW88wbCgUemjiRpLVlKJamJiXleWAl23iejgAvmFT7mz4OF8wsF83+FDJo3h2YVhVJYXg5lzaCyReHzrKxwHeWK4wMrWmJeW3jMvLZQEGs/ub3sY1hS+8n9/77vv47La+v8e1/yv3/RrKLwfVW2gJatoH1HaF4FrdsUimirNtC8hSOgUpEJPTcnDNydOPpW4oBdCZ27po4kaS1YSiWpEYu1y2HOLJg9E+bMLFwPOXd2YbTz3yoqoVVraFcDXbsV9s9s3uI/H5XNoaKywYtZjLFQXmuX/6ek1tZ+9vPa5Z8e2fxfK0p0GVWtWrNo6ceF2xUVhDJfBqVSFQ46ivj80+TXXkx26m/85ZFUwnw1lqRGJC5fDjOmwbQP4MMPCovxxE9GPptVFIpnz80Lf7ZpC9Wti/Z6rBDCf6bn1pGy6mrCggV19niS0gmt2hAOPIJ49YXEpx4mDBicOpKkNWQplaQSFmMsFM8p78K0KTBzWmF0MYTCVNTeWxf+bFdTKKCOJEhqRMJOQ4mPjiH+4wriVv0IVdWpI0laA5ZSSSoxMc/hwynw/lvw3tuweGGhhLatgV5bQacu0GE9QkVF6qiSVK9ClhUWPfrNKcTb/k4YcVzqSJLWgKVUkkpAzHOY+j68+ya8/zZ8vLRwrWSXrp/sjblh0U7DlaT6FDbsThi8J/HBu4gDdyds2D11JEmryVIqSUUszpsLb7wCb00qbNXSrAI22Ag26AadNyCUN0sdUZKSC/t/nfjsY+TXXkR25tmELEsdSdJqsJRKUpGJy5bBe28Wyuj0qYWpuV02hE02g85dCXW48I8kNQahqppw8DeJl/+B+Oh9hEHDUkeStBospZJUJOKCefDKC/DmK7BsWWGPzD4DoNumhKqWqeNJUlEL/QcTH7mP+M+/EvsMILRqkzqSpFVkKZWkxOKs6fDyc4XRUQJs2L2wbUvH9VwtV5JWUQiB7OvHk//y+8R//pVw1EmpI0laRZZSSUogxggfvAsTnyuspNusAjbbGnptRWjplgaStCZC566EIfsR772ZuNMehE02Sx1J0iqwlEpSA4oxFlbPff5pmDsbqqqh70DYpLdbuEhSHQj7HEp8emxh0aOf/MHr8KUSYCmVpAYQYyxs6fLcUzBrBrReB3YcAht1J2S+YZKkuhKatyA77Bjyi0YSH7yTMGTf1JEkrYSlVJLqWfxwSqGMTp8KLVvBwN1g455uWSBJ9aXPDrBFX+Jt1xL77UhYp33qRJK+hO+IJKmexDmziPffDqNvhfkfwfaDYL+vEbr3spBKUj0KIZCNOA6WLyfeeEXqOJJWwpFSSapjcemSwjWjr71cWMBo2x1g0y0I5c1SR5OkJiN0XI+w50HE268j7jyUsNnWqSNJ+gL+ql6S6kjMc+Kkl+DWawuFtOfmsP/XCZv3sZBKUgJhz69Ch07kf7+YuGxZ6jiSvoClVJLqQJz2Adx5Izw9FtrWwN6HELYfRKhsnjqaJDVZoVkF2Yhvw7QPiPfdmjqOpC/g9F1JWgtx8SIY9xi883phEaNBw6BrN0IIqaNJkoCwZV/YdiDxzhuI2w8i1KybOpKk/+FIqSStgRgj8c1X4V/XwXtvwlb9YN8RhA27W0glqchkh34LQkZ+/aWpo0j6HI6UStJqivPnwVMPwdTJ0KETDBhMWKdd6liSpC8Q2nUgfOUw4k1XEZ9/mrD19qkjSfovllJJWkUxz+HVF+C5pyELhS1eem7uyKgklYCw+77Ex+4nv+4Ssl5bEyorU0eS9Amn70rSKogfzYF7boZnH4dOXeArhxE23cJCKkklIpSXk339BJg1nXjnDanjSPovjpRK0peIMcKkl2D841DeDHbeAzbcxDIqSSUobLoFYYfdiKNvIfYfTOilxucTAAAgAElEQVTSNXUkSThSKklfKC5aAPffAc88Uhgd3edQwkY9LKSSVMLCwUdD8yryay4sXJYhKTlLqSR9jvjO63D7DTB9KvTfBXbdm1DVMnUsSdJaCq1aEw46Ct6YSHz8/tRxJOH0XUn6lPjxUnhqbGHf0Zp1YcfdCa3XSR1LklSHwsDdiY/dX1iNd+vtCa3apI4kNWmOlErSJ+LMD+HOf8C7b8DW28OwAyykktQIhSwjO/xEWLKI+I8rU8eRmjxLqaQmL8ZInPg83HsLxLxQRrfqR8j8ESlJjVXo0pUw9ADiEw8QJ72YOo7UpPmOS1KTFpcugYfuhmcfg84bwt6HEDp0Sh1LktQAwt6HQs265NdcRFy2LHUcqcmylEpqsuL0qXDHjTDlPei3IwweTqhsnjqWJKmBhMpKsq8fD9MmE++9OXUcqcmylEpqcmKMxJcnwOhbIctg+IGEzbZ2qxdJaoLCFn0JfXck3nkjcfqU1HGkJmmlq+9eeOGFjB8/njZt2jBq1KjP3B9j5Morr2TChAlUVlZy4okn0q1bt3oJK0lrKy5eBA/fA++/DV27ww6DCRWVqWNJkhIKhx1DnDiB/JqLyH7wS39JKTWwlY6UDh48mB/96EdfeP+ECROYNm0a559/PscddxyXXXZZnQaUpLoSp75P/ttTYPI70HcgDBpqIZUkEdZpTzjgG/DK88QnHkwdR2pyVlpKe/fuTXV19RfeP27cOAYNGkQIgZ49e7Jw4ULmzJlTpyElaW3FZx8n/82psHABDNmX0HsbfxMuSVoh7DIcuvci3ng5cd7c1HGkJmWtrymdPXs2NTU1K263b9+e2bNnr+3DSlKdiLW15DddSX7xSOjSlewnfyB06pI6liSpyIQsI/vGd2HJYuINl6eOIzUpK72mdGVijJ/5uy8afRgzZgxjxowBYOTIkZ8qs2o8ysvLfW5VZ9bmfMrnzeWjc/8fH7/4LC2GH0Cro79PaFbBondereOUKhVlWdmXzv6RVkepnE9VviavupoaFhz0DRbecAWthu5LZd8dGuxL+/5JdanUzqe1LqXt27dn5syZK27PmjWLtm3bfu6xQ4YMYciQIStu//e/U+NRU1Pjc6s6s6bnU5z8Dvmffw0fzSEc9X0+3nF3Zn00D4B8wYK6jqkSUV1dzQKff9WRUjmfFvmavFriLnvD2PuYe9HvyH7xJ0LzFg3ydX3/pLpULOdT586dV+m4tZ6+269fP8aOHUuMkddee42qqqovLKWS1BDihCfJR54Oy5eTnf5/ZDvunjqSJKlEhGbNyI74DsyaTrzt76njSE3CSkdKzzvvPCZOnMj8+fM5/vjjOeSQQ1i+fDkAQ4cOpU+fPowfP56TTjqJiooKTjzxxHoPLUmfJ8ZY2Gfutmthox5k3/kRYZ32qWNJkkpM6NGbsMtw4v23E7cfRNi4R+pIUqO20lJ68sknf+n9IQSOOeaYOgskSWsiLl1KvOqPxHGPEgYMJhzxHbd7kSStsXDgkcTnnyb/25/JfjyKUL7WV71J+gJrPX1XklKLs2eQn30G8dnHCAcdRTj6BxZSSdJaCVUtyUZ8Gya/TbzvttRxpEbNUiqppMW3Xyf/7akwYxrZ935KNuxA9x+VJNWJsO0O0GcA8fbriNM+SB1HarQspZJKVnz2MfJzzoLyZmRnnE3Ysl/qSJKkRib72vHQrBn5X/9EzPPUcaRGyVIqqeTEGMnvvJH84t9B125kPzqX0KVr6liSpEYorNOOcOgx8MZE4oN3po4jNUqWUkklJS5bRrzyj8RbryFsP4jslF8TWq+TOpYkqRELO+wGW/Yj3vw34vSpqeNIjY6lVFLJiPPnkf/hp8QnHiB8ZQThmFMIzSpSx5IkNXIhBLLDT4SyMvK//dlpvFIds5RKKgnxwynkI0+Dt18nHHMK2b4jXNBIktRgQrsawsFHw6QXiWPvSR1HalQspZKKXnzjlUIhXbSA7JRfk/XfJXUkSVITFHbaA3r3Id70V+Ks6anjSI2GpVRSUVvyxIPko34CVdVkZ55D2GSz1JEkSU1UCIHsG98FKEzjjTFxIqlxsJRKKkoxRvL7buOjc35SWGH3zLMJ63ZOHUuS1MSF9h0IBx0FE58jPnpf6jhSo2AplVR0Yl5LvP5S4o2XU9l/l8IKu63apI4lSRIAYdAw2HRL4j+uIM6ekTqOVPIspZKKSly6lPyikcQH7iAM2Y82p/6KUFGZOpYkSSuELCM78nuQ5+R//ZOr8UpryVIqqWjEeXPJR/0Ynn+acNixZId+i1BWljqWJEmfETp0KqzGO/E54sN3p44jlTRLqaSiEKdNJh95Okx+h+z4M8l2/0rqSJIkfakwaBhs0Zd405XEaZNTx5FKlqVUUnLx9YnkI8+AxYsK149uu0PqSJIkrVQIoTCNt1kl+RXnEWtrU0eSSpKlVFJScdyj5L//KbRsRXbWOYTuvVJHkiRplYV12pEdfgK8/Rrx7n+kjiOVJEuppCRijOT33kL+l7Nhw+6FLV86rpc6liRJqy3024mw/S7EO24gvvtG6jhSybGUSmpwMa8lXvcX4k1XQt+BZD/8FaFV69SxJElaY+Fr34ZW65Bf/gfix0tTx5FKiqVUUoNaseXLg3cRhu5PdtzpbvkiSSp5oWU12TdPgqnvE2+5OnUcqaRYSiU1mDj/o/9s+TLiOLKDjyZk/hiSJDUOoXcfwq57E8f8i/jK86njSCXDd4OSGkT8cAr5/532ny1fdtsndSRJkupc+OpRsG4X8iv/SFw4P3UcqSRYSiXVu/jmq+QjT4PFC93yRZLUqIXKSrJjT4F5c8n/+idijKkjSUXPUiqpXsXxT5CP+gm0aEl2plu+SJIav7DhJoQDvwETniQ+fHfqOFLRs5RKqjf5/XeQXzwS1t+osOXLup1TR5IkqUGEIfvCFtsSb7icOPmd1HGkomYplVTnYp6T/+MK4vWXwNbbk53yG0LrdVLHkiSpwYQsI/vm96GqJfml5xKXuk2M9EUspZLqVFz2MfHSc4mjbyXsuhfZCWcSKt3yRZLU9ITWbcm+9QOY8h7xH5enjiMVLUuppDoTF84n//3PiOMeJRx0FGHEtwlZWepYkiQlE3r3IQw7kPjwPcTxj6eOIxUlS6mkOhFnfkg+8gx45zXCsaeSDTuQEELqWJIkJRf2/zps1KOwGu+sGanjSEXHUipprcV33yjsQTpvDtkPfkm2/aDUkSRJKhqhvBnZsadCnpNfNopYW5s6klRULKWS1kp8cRz5OT+C8mZkZ/yO0HOL1JEkSSo6oeN6hMNPhDcmEm+9JnUcqaiUpw4gqXTlY+8lXntRYcuX7/2MsE671JEkSSpaWf9dyF9/mXjPP4ndNiX0GZA6klQUHCmVtNpijOS3XEO8+gLovQ3Zab+1kEqStArCocfChpuQX3kecfqU1HGkomAplbRa4vJlxCvOI951I2GnPci+8xNC86rUsSRJKgmhWTOyE86ErIz8opHuXyphKZW0GuKiheTn/5L45IOE/b5G+MZ3CeVeBSBJ0uoI7TuSHfND+OBd4rUXEmNMHUlKylIqaZXEWdPJzz4TXnuJ8M3vk+1zmFu+SJK0hsIWfQn7HEZ84kHiI/emjiMl5RCHpJWKb75KfsFvYPlyspN+RujdJ3UkSZJKXtjnUOLbk4jXXcKyLbeFth1TR5KScKRU0pfKnx5Lfu6PoXkLsrPOtpBKklRHQpaRfeuH0Lotc8/5MXHBvNSRpCQspZI+V4yR/F/XES89FzbqQXbWuYT1NkgdS5KkRiVUtyY7/kzyObPJLzmHWFubOpLU4Cylkj4jLvuYeNko4u3XEXbYleyHvyK0ap06liRJjVLYuAetTzgdXnmeeOPlqeNIDc5rSiV9Spw3h/zC/4M3XyUccARhz4Nc0EiSpHrWYre9mT/pZeLoW8k7dyXbZXjqSFKDsZRKWiF+8C75n34F8+eSHX8Goe+OqSNJktRkhK8eSZzyPvG6vxA7dSFsumXqSFKDcPquJADii8+Sjzwdli8jO/X/LKSSJDWwkJWRHXsqdOxMftFI4oxpqSNJDcJSKon8/jsKI6QdOpH96FzCxj1SR5IkqUkKVS3JvvtjAPI//Yq4eFHiRFL9s5RKTVisrSX/+8XE6y+BrfqRnT6S0K5D6liSJDVpoWNnsuPPgOlTyC89l5i7Iq8aN0up1ETFRQvJ//RL4oN3EYYeQHbiWYTmLVLHkiRJQOi1FeGw4+DFccR//i11HKleudCR1ATFaR+QX/AbmDGV8I3vku08NHUkSZL0P7LBe5JPeZc4+hby9h3IdtsndSSpXqxSKX3uuee48soryfOc3Xffnf333/9T9z/00ENcffXVtGvXDoDhw4ez++67131aSWstvjiO/NJzoayc7Ae/dGU/SZKKWDj0WOLsmcTrLyW2aUfoOzB1JKnOrbSU5nnO5Zdfzk9+8hPat2/PWWedRb9+/Vh//fU/ddzAgQP51re+VW9BJa2dGCPx7puIt14DG2xMduKPCO07po4lSZK+RCgrIzv2NPI//JT8slFkrdoQem6eOpZUp1Z6Tekbb7xBp06dWHfddSkvL2fgwIE888wzDZFNUh2JS5cQ/3I28ZarCdvtTHb67yykkiSViFBZSfa9n0LNuuQX/Jr4wXupI0l1aqWldPbs2bRv337F7fbt2zN79uzPHPfUU09x6qmnMmrUKGbOnFm3KSWtsThjGvnI04njnyAcdBThmFMIlZWpY0mSpNUQWrYiO/kXUFFJ/sdfEGfPSB1JqjMrnb4bY/zM34UQPnW7b9++7LjjjjRr1ozRo0dzwQUX8POf//wz/27MmDGMGTMGgJEjR1JTU7OmuVXEysvLfW6LxNIJT/LR739BiJE2Pz2Xyj4DUkdabfVxPi2qrq7Tx1PpKMvKqPb5Vx0plfOpytfkkrBKr3c1NSz7+XnM+fEJhD//mna/vYisunXDBFRJKbX34ystpe3bt2fWrFkrbs+aNYu2bdt+6phWrVqt+HzIkCFce+21n/tYQ4YMYciQIStuO6LaONXU1PjcJhbznHjHDcQ7rocuG5KdcCbzO3Zmfgk+L/VxPuULFtTp46l0VFdXs8DnX3WkVM6nRSX4s78pWuXXu+p1CCecRe0ff8GMX/6Q7OT/R6hwBpQ+rVjej3fu3HmVjlvp9N3u3bszdepUpk+fzvLly3n88cfp16/fp46ZM2fOis/HjRv3mUWQJDWcuGBeYf/R268jDBhMduY5hI6r9gNBkiQVv9BrK8LRP4Q3XiG/+HfEZctSR5LWykpHSsvKyjj66KP5zW9+Q57n7LrrrmywwQbccMMNdO/enX79+nH33Xczbtw4ysoK01hOPPHEhsgu6X/Ed14nv/h38NFswuEnEgYN+8x0e0mSVPqy7XYiX7yQePUF5JecTfbtMwjlq7Tbo1R0Qvy8i0YbyJQpU1J9adWjYpku0JTEGImP3Eu87hJo3Zbs+DMJG/dIHatO1Mv03bH31OnjqXSUynRLlYZSOZ+yQcNTR9AqWNPXu/zBO4l//wuh746EY08llJXVQzqVmmJ5P76q03f9dYpU4uKSRcRrLyY++RBs3ofsmFMILnogSVKTkO26N/ny5cQbL4eycvjWyYTMYqrSYimVSlh87y3yv5wNM6YR9v0aYe+DfSGSJKmJyfbYj3z5MuLNf4Pycjjye4RspUvHSEXDUiqVoBgj8cE7if+4Aqpbk53ya8KmW6SOJUmSEsn2PIh82TLi7ddBeTM4/ATXlVDJsJRKJSYuXED+1/NhwpOwZT+yb55MaOV0XUmSmrrwlcNg+TLi3TdBlsGI4xwxVUmwlEolJL75Kvml58LcWYSDjyYM2dcXG0mSBFAYGT3gCMhriffeAksWw1EnufiRip6lVCoBcfly4p03Eu+8Edp3IDvjd4SNe6aOJUmSikwIAb56FLRoSbz1GuKSxWTHnUpoVpE6mvSFLKVSkYvTPiC//PfwzuuEAbsSRhxHqGqZOpYkSSpSIQTC3oeQt6giXncJ+fm/JPvOjwnNW6SOJn0uS6lUpGKMxIfvKSxmVN6M7NunE/rtlDqWJEkqEdlu+5A3ryL+9Xzy3/+U7Ps/J7RslTqW9BmWUqkIxXlzyK/6E7w4DnpvQ3bU9wlt26eOJUmSSkw2cDdiiyryS84mP+dHZD/4JaFN29SxpE9xhRSpyMRnHyP/xUnw6guEw44l+/4vLKSSJGmNhT4DyL73M5j5IfnvziB+OCV1JOlTLKVSkYjz5lB70Ujyi38H7TqQ/fj3ZLt/xdV1JUnSWgu9tyH7wS9h8SLy/zuNOOml1JGkFXy3KyUWYyR/6mHyn38XXniacOA3yM46h9Cla+pokiSpEQnde5GddQ60akP+h5+RP3Z/6kgS4DWlUlJx7izyay6C55+GbpuSHXUSYb0NUseSJEmNVOi4HtlZZ5Nf/DviVX8k/3AyYf8jnJmlpCylUgIxz4mP319YWXfZMsLB3yQM2ZeQubm1JEmqX6GqmuyknxOv+wvx7n8SP5xKdvQPCJWVqaOpibKUSg0sfvBuYXT0jYmwSW+yI79H6NQldSxJktSEhPJyOPxEWLcL8aYryc+ZTvbdHxPWcXFFNTxLqdRA4tIlxNuvJ465DVpUEY78HmHg7k6XkSRJSYQQCEP3J67bmfzSc8l/9QOyY08l9NoqdTQ1Mb4blhpAfO4p8p99h3jvzYQBu5L98iKynfawkEqSpOTC1tuTnXUuVFWT//5n5HfeSMzz1LHUhDhSKtWjOH0q+Y2XFxYy6tyV7PSRhB69U8eSJEn6lNClK9mPRxGvvoB46zXEN18lO/pkQnXr1NHUBFhKpXoQFy8i3nkj8f5/QVk54cAjCXvsV7h+Q5IkqQiF5i3gmFOgR2/iDZcVpvN++3RCt01TR1Mj5ztkqQ7FvJb42P3EW66G+R8Vrhk94HAXDZAkSSUhhEAYvBdxox7kF/+O/OyzCAcdRdhtHy87Ur2xlEp1JE56ifyGS+H9t2GTzchO+hlhox6pY0mSJK22sFEPsp+eR37lecQbLiO+8AzZkScR2ndIHU2NkKVUWktx8jvkt15TuG60XQfCcacR+u1ECCF1NEmSpDUWWlaTfefHxEfuJd54Bfn/+x7h0GMJA3fzfY7qlKVUWkNxxjTiv/5OfOphaF5F2P/wwnWjFW48LUmSGocQAmHQcOJm25Bf9UfiVX8kTniC7IjvENq0TR1PjYSlVFpNce7swiJGj4yGLCMMO5Aw/EBCy1apo0mSJNWL0KET2Sm/Id5/O/Hmv5H/4rtkXz+B0G+n1NHUCFhKpVUU580l3ncb8YE7YPkyws5DCfsc6iJGkiSpSQhZRthjP+IW25JfcR75X86GJx8iO/QYQodOqeOphFlKpZWIc2YRR99CHHsPLFtG2G5nwn5fI3TsnDqaJElSgwvrbUB25tnEMbcRb7+e/OffJex1cGH2WLNmqeOpBFlKpS8QZ35IvPdm4qP3QZ4T+g8m7HUQodP6qaNJkiQlFcrKCMMOJG43iHjj5cTbriU+8QDZiG8Tttg2dTyVGEup9D/iB+8S77uV+ORDQCDsOKRwzajTUiRJkj4ltKshHH8G8eUJ5NddQv7HX8C2O5Ad8i1C+46p46lEWEolIOY5vDyBfMxtMPE5qKggDN6LMPQAQrua1PEkSZKKWti8D9nPzy9c8nTXjeQvjCPsuhdhz4MJrVqnjqciZylVkxY/Xkp88kHimNth6vvQpl1ha5ddhhOq/QEqSZK0qkKzZoS9DyHusCvxX9cRx9xOfGQ0YdgBhCH7EZq3SB1RRcpSqiYpTp9KfGQ08dHRsGA+dO1G+NYPCP12IpR7gb4kSdKaCu06EI46iTjsAPJbryHe9nfiA3cS9j6UMGiYiyHpMyylajLi8mXw3FPkY++FV56HLIOtticbsi/03JwQQuqIkiRJjUZYbwPKTjiL+NYk8pv/Rrz+EuJ9txKG7k/YcQ9CZWXqiCoSllI1enH6VOKjo4mPjoH5H0G7DoUtXXbcg9DWPUYlSZLqU+i2Kdkpv4ZXniP/13XE6y4h3n4dYdd9CLvt7SVTspSqcYoL5hHHPUZ88kF489VPRkW3Ixs0DDbvQ8jKUkeUJElqMkII0LsPZb37EF+fSH7PP4m3X0e892bCzkMJe+znar1NmKVUjUZc9jG8MI78yQfhxWehdjl07ko48EjCgMGOikqSJBWB0KM3ZT16F7bhu/dm4kN3ER+8E/oMIBs0HHptRciy1DHVgCylKmnx46UwcQJx/BPE556GxQuhTdvCVJABu8IGG3utqCRJUhEKXTYkHP0D4v6HE++/nfjY/eTPPg4d1yssiDRwd0KrNqljqgFYSlVy4pJFxBefhWcfJ770LCxdAlXVhG36E/rvAptt5fRcSZKkEhHadSAcfHShnD77GPHhe4k3XUW89RpCnx0IOw+FTbfw/V0jZilVSYgzphFferZQRl95HpYvg1ZtCP0HE/ruAD23JJR7OkuSJJWq0KyiMNNtwK7ED94jPnIv8YkHiM88UthLfrudCNvtDBv3dCZcI+O7eBWluOxjeO3l/xTRDz8o3NFxPcIu/7+9e42Nqtz3OP5dndKW0vv0Rsul0F0wgCIwJQpbAkLk7POCsEnQHW8viDtRxIBEULwbQ2iMEhPAlBhscMcXhJx4PMeccziB6i6hGzdYy0XlUlqg0JEyHaBlSrHtPPvF6oVKywxaZjH190kms2bW6uQ/8J/Lb55nrfVvWNNnwR/u0S9mIiIiIkOQlT8G6y9/xfz5aczhA5h/Vtj7nu7+L8jKxSp+CGvmHPv4IQqoUU+hVO4KpqMDTp/EHD+COX4ETv0IP/8MscPgnnux5v071pQZWDl5TpcqIiIiIhFixcdjFf8Riv+Iab2K+W6/HVD/9z8w/7PTDqj3FWPdV2yfdz52mNMly6+gUCqOMO0/w5kazMkfMccPQ82P9r6hAPljsR5aiDV5mj0tVydWFhEREfndsxKTsGYvgNkLMM2XMFX77VHUil2YPf8NCcNh0jQ7pE6ehpWW4XTJEiaFUrnjjDHgv4g5dQxqj2Nqj8PZWvuULQAjR2PNehhr4n32L1w6ypqIiIiI3IKVko41908w90+Y69fh2GE7oB4+gKmqxADk5mNNuNeedTdhClZqutNlywAUSmVQGWPovNCAOXQQc6YWc/YU1NfClUv2BnFxUFBknyC5cCKMn4iVojcIEREREfl1rPh4mFqMNbXYHgypr8McO4Q5dgTzz79Dxf/ZIXXkaKyiyfb3z3FFkDtK50O9SyiUyq9mAi1w/izGWw8NZzHnz0B9Lb7WgL1BTIz94p80DQr+gFV4D+QX6Ci5IiIiInJHWJYFY8ZjjRkPj/wZ09kJZ2sxJ47YIfVARW9ITRhuD5aMK8IaNxFGjwN3tg6c5AClA7kl09EOvgvQ6MU0NsAFL+anc+Ct7x39BIgfDnmjsTwPkTTpPgIZOZA/BitO+4OKiIiIiDMslwvG2cGThUswwSBcOI+pOwF1JzF1JzD//592eAU7qOaPxcovgFFd1/ljsEYkO/k0hjyF0t85E+yEy35ouohpaoSui2m6CBe94GsEE+z9g+GJkJOPNXm6fQjuvNGQNwbSM3umPyRmZtLq8zn0jERERERE+md1z+QbORpmzQe6DsB5thZz/jScO405fwZzcG/viCrAiGT71IQ5+ZAzErLz7LNCuLNhRLJGV38jhdIhyBgDbdfgajNcbYFAM+bKZbjcBFf8mMt+O4he8dujnd2/DHVLToWMLKyCIpg5x37RZY+EnDxIStGLTkRERESGDGtYHBTeY+9q1sUYA5ea4PxpjPccXGjANDbYpy7c/5W9TffG8QmQkQXuLKyMLHs53Y2VmgGpaZCaYQdX7b86oLBCaXV1NWVlZQSDQebPn8/ixYv7rG9vb2fz5s3U1taSnJzMqlWryM7OviMF/x4YY6D9Z/sUKd2Xtms9y+ZawA6bV5sh0ILpDp9dt7na0ntk219KTIK0DEjLwModBenurhdQtv1LT0aWTsEiIiIiIr9rlmVBRiZkZGLd6+mzzly/bs8obGzA+C/2zjj0+zCna+zv5NwQWgFcLkhOg5Q0SEq2pwMnJcOIlK7rZKzhifb04fgEe9e47uWEBKwYV+SevANChtJgMMi2bdt4/fXXcbvdrFu3Do/Hw6hRo3q2KS8vZ8SIEWzatIl9+/bx2Wef8eKLL97Rwu+04M4yCHaCZdkXrN7lX15uXNfZYY883njd0btsblrX3hU2r8P1ruDZ1tZ3yuxAYmLsqQRJXc2ck4fVvTwixR7V7GpyUtMhNV37eIqIiIiI/AZWfDyMKoBRBfQ3f9Bcv947I7H5EubKJXv5yiVM82V7UMnXaIfXawEwdnw1/TxWj7i4vkE1PgEShhPzxHNYWbl34FlGVshQWlNTQ25uLjk5OQDMmjWLAwcO9AmlBw8eZOnSpQA88MADfPLJJxhjonqap/lHuT1aaegKiKZ3+cZrTE8jAXYwdbnAFXvDdWw/97kgNhZih0GaG6u7uXp+GUmAuK5fRrrv614/fLgdRIePiOp/YxERERGRocaKj4fskfYF+g2u3UywEwIBCDT3zoxsu4Zpu2YPWLXdOGvymr2ueyZla6BrgCz6hQylfr8ft9vdc9vtdnPy5MkBt3G5XCQmJtLS0kJKSsoglxs5ro1/u63tjTFggkN+aF1ERERERAaHFeOC5BT7cuP9DtXjlJCh1JibB5J/OToXzjYAu3fvZvfu3QCUlJSQl5cXdqESXfR/K4Np0PvpL8sG9/EkqqQ5XYAMKeonGUz6/iSDKZr6KeQhoNxuN01NTT23m5qaSE9PH3Cbzs5OWltbSUpKuumxFixYQElJCSUlJb+1brmLvfLKK06XIEOI+kkGk/pJBpP6SQaT+kkGU7T1U8hQWlhYiNfrpbGxkbgmsiQAAAYASURBVI6ODiorK/F4+h6BasaMGXz99dcA7N+/n8mTJ2tfRxEREREREQkp5PRdl8vFsmXLWL9+PcFgkHnz5jF69Gh27NhBYWEhHo+Hhx9+mM2bN/PCCy+QlJTEqlWrIlG7iIiIiIiIRLmwzlM6ffp0pk+f3ue+xx57rGc5Li6O1atXD25lErUWLFjgdAkyhKifZDCpn2QwqZ9kMKmfZDBFWz9Zpr+jFImIiIiIiIhEQMh9SkVERERERETulLCm74r8UnV1NWVlZQSDQebPn8/ixYv7rP/yyy/Zs2cPLpeLlJQUnnvuObKyshyqVqJBqJ7qtn//fjZu3MiGDRsoLCyMcJUSLcLpp8rKSnbu3IllWYwdO5aVK1c6UKlEg1D95PP52LJlC4FAgGAwyOOPP37Tbk8iAB999BFVVVWkpqbywQcf3LTeGENZWRnfffcd8fHxLF++nPHjxztQqUSDUP20d+9evvjiCwASEhJ45plnKCgoiHCVYTIit6mzs9OsWLHC/PTTT6a9vd289NJLpr6+vs82R44cMW1tbcYYY3bt2mU2btzoRKkSJcLpKWOMaW1tNW+++aZ59dVXTU1NjQOVSjQIp58aGhrMmjVrTEtLizHGmMuXLztRqkSBcPqptLTU7Nq1yxhjTH19vVm+fLkTpUoU+P77782pU6fM6tWr+13/7bffmvXr15tgMGiOHz9u1q1bF+EKJZqE6qdjx471fM5VVVXd1f2k6bty22pqasjNzSUnJ4fY2FhmzZrFgQMH+mwzZcoU4uPjASgqKsLv9ztRqkSJcHoKYMeOHSxatIhhw4Y5UKVEi3D6ac+ePSxcuLDnnNqpqalOlCpRIJx+siyL1tZWAFpbW286n7tIt0mTJvW87/Tn4MGDzJkzB8uymDBhAoFAgEuXLkWwQokmofpp4sSJPeuLiopoamqKVGm3TaFUbpvf78ftdvfcdrvdtwyd5eXl3H///ZEoTaJUOD1VV1eHz+djxowZkS5Pokw4/dTQ0IDX6+WNN97gtddeo7q6OtJlSpQIp5+WLl3K3r17efbZZ9mwYQPLli2LdJkyRPj9fjIzM3tuh/qOJRKu8vJypk2b5nQZA1Ioldtm+jlgs2VZ/W5bUVFBbW0tixYtutNlSRQL1VPBYJDt27fz9NNPR7IsiVLhvEcFg0G8Xi9vvfUWK1eupLS0lEAgEKkSJYqE00/79u1j7ty5lJaWsm7dOjZt2kQwGIxUiTKE3M53LJFwHT16lK+++oonnnjC6VIGpFAqt83tdvcZ/m9qaup3qtLhw4f5/PPPWbt2raZbyi2F6qm2tjbq6+t55513eP755zl58iTvvfcep06dcqJcucuF8x6VkZFBcXExsbGxZGdnk5eXh9frjXSpEgXC6afy8nIefPBBACZMmEB7ezstLS0RrVOGBrfbjc/n67k90HcskXCdOXOGrVu3smbNGpKTk50uZ0AKpXLbCgsL8Xq9NDY20tHRQWVlJR6Pp882dXV1fPzxx6xdu1b7aklIoXoqMTGRbdu2sWXLFrZs2UJRURFr167V0XelX+G8R82cOZOjR48C0NzcjNfrJScnx4ly5S4XTj9lZmb29NO5c+dob28nJSXFiXIlynk8HioqKjDGcOLECRITExVK5Vfz+Xy8//77rFixgry8PKfLuSXL9DdPQCSEqqoqtm/fTjAYZN68eSxZsoQdO3ZQWFiIx+Ph3Xff5ezZs6SlpQH2B/bLL7/scNVyNwvVUzd6++23eeqppxRKZUCh+skYw6effkp1dTUxMTEsWbKE2bNnO1223KVC9dO5c+fYunUrbW1tADz55JNMnTrV4arlbvThhx/yww8/0NLSQmpqKo8++igdHR0APPLIIxhj2LZtG4cOHSIuLo7ly5frs04GFKqfSktL+eabb3r2U3a5XJSUlDhZ8oAUSkVERERERMQxmr4rIiIiIiIijlEoFREREREREccolIqIiIiIiIhjFEpFRERERETEMQqlIiIiIiIi4hiFUhEREREREXGMQqmIiIiIiIg4RqFUREREREREHPMvEmDS6+DcJdoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Just for fun\n",
    "sns.distplot(cvd);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.4\n",
    "\n",
    "In this chapter we have focused on one reason why k-fold CV fails in financial applications, namely the fact that some information from the testing set leaks into the training set. Can you think of a second reason for CV's failure?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Overfitting due to multiple train/test bias (explored further in chapters 11-13)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7.5\n",
    "\n",
    "Suppose you try one thousand configurations of the same investment strategy, and perform CV on each of them. Some results are guaranteed to look good, just by sheer luck. If you only publish those positive, and hide the rest, your audience will not be able to deduce that these results are false positives, a statistical fluke. This phenomenon is called \"selection bias.\"\n",
    "\n",
    "- a) Can you imagine one procedure to prevent this?\n",
    "\n",
    "- **A: Publish the amount of trials it took to get the results**\n",
    "\n",
    "- b) What if we split the dataset in three sets: training, validation, and testing? The validation set is used to evaluate the trained parameters, and the testing is run only on the one configuration chosen in the validation phase. In what case does this procedure still fail?\n",
    "\n",
    "- **A: If repeatedly test against the 3rd set, we can still bias our results.**\n",
    "\n",
    "- c) What is the key to avoiding selection bias?\n",
    "\n",
    "- **A: Taking the number of trials into account and not using backtests as a research tool.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}