notebooks/11_cross_validation.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:4aa33b85c25916966326c2f6ba5b3a06c4df0e8ad5ba66461d3b5f66460f3892"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Cross-Validation"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.datasets import load_iris\n",
      "from sklearn.cross_validation import train_test_split, cross_val_score\n",
      "from sklearn.neighbors import KNeighborsClassifier\n",
      "from sklearn.linear_model import LogisticRegression, LinearRegression\n",
      "from sklearn import metrics\n",
      "import pandas as pd\n",
      "import numpy as np\n",
      "import matplotlib.pyplot as plt\n",
      "%matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## What's Wrong with Train/Test Split for Model Evaluation?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Previously, we studied the **train/test split** procedure for model evaluation. We found that it was a superior alternative to training and testing on the same data because it helps to avoid overfitting.\n",
      "\n",
      "However, we also found that it provides a **high variance estimate** of out-of-sample accuracy, since changing which observations happen to be in the training set versus the testing set can make a meaningful difference in the estimated accuracy.\n",
      "\n",
      "Let's see this in code:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# load the iris data\n",
      "iris = load_iris()\n",
      "X = iris.data\n",
      "y = iris.target"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# use train/test split with random_state=4\n",
      "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=4)\n",
      "\n",
      "# check accuracy of KNN with K=5\n",
      "knn = KNeighborsClassifier(n_neighbors=5)\n",
      "knn.fit(X_train, y_train)\n",
      "y_pred = knn.predict(X_test)\n",
      "print metrics.accuracy_score(y_test, y_pred)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.973684210526\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# use train/test split with random_state=1\n",
      "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
      "\n",
      "# check accuracy of KNN with K=5\n",
      "knn = KNeighborsClassifier(n_neighbors=5)\n",
      "knn.fit(X_train, y_train)\n",
      "y_pred = knn.predict(X_test)\n",
      "print metrics.accuracy_score(y_test, y_pred)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1.0\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "One **solution** to this problem might be to take a bunch of train/test splits, and then average the results together. That's the essense of cross-validation!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Steps for K-fold Cross-Validation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "1. Randomly split the dataset into K equal partitions.\n",
      "2. Use partition 1 as the test set and the union of the other partitions as the training set.\n",
      "3. Calculate test set accuracy.\n",
      "4. Repeat steps 2 and 3 K times, using a different partition as the test set each time.\n",
      "5. Use the average test set accuracy as the estimate of out-of-sample accuracy.\n",
      "\n",
      "Here are two diagrams of **5-fold cross-validation:**"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "![5-fold cross-validation](images/cross_validation_diagram.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "![5-fold cross-validation](images/cross_validation_example.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Cross-Validation Recommendations"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Any number of folds can be used, but **10-fold cross-validation** is generally recommended because it has been shown experimentally to produce the most reliable estimates of out-of-sample accuracy.\n",
      "\n",
      "For classification problems, **stratified sampling** is also recommended, meaning that each response class should be represented with approximately equal proportions in each of the K folds. (scikit-learn does by default when using `cross_val_score`, which we will use below.)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Comparing Cross-Validation to Train/Test Split"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Advantages of **cross-validation:**\n",
      "\n",
      "- More accurate estimate of out-of-sample accuracy\n",
      "- More efficient use of data (every record is used for both training and testing)\n",
      "\n",
      "Advantages of **train/test split:**\n",
      "\n",
      "- Runs 10 times faster than 10-fold cross-validation\n",
      "- Simpler to examine the test set results without writing custom code"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Cross-Validation Examples"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's try out cross-validation on a few datasets:\n",
      "\n",
      "- **iris:** multi-class classification\n",
      "- **default:** binary classification\n",
      "- **advertising:** regression"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Iris dataset\n",
      "\n",
      "- Multi-class classification\n",
      "- Using accuracy as evaluation metric\n",
      "- Using cross-validation to select tuning parameters (aka \"hyperparameters\")"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# load the iris data\n",
      "iris = load_iris()\n",
      "X = iris.data\n",
      "y = iris.target"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# 10-fold cross-validation with K=5\n",
      "knn = KNeighborsClassifier(n_neighbors=5)\n",
      "scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')\n",
      "print scores"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[ 1.          0.93333333  1.          1.          0.86666667  0.93333333\n",
        "  0.93333333  1.          1.          1.        ]\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# use average accuracy as estimate of out-of-sample accuracy\n",
      "print np.mean(scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.966666666667\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# search for an optimal value of K\n",
      "k_range = range(1, 30)\n",
      "scores = []\n",
      "for k in k_range:\n",
      "    knn = KNeighborsClassifier(n_neighbors=k)\n",
      "    scores.append(np.mean(cross_val_score(knn, X, y, cv=10, scoring='accuracy')))\n",
      "print scores"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[0.95999999999999996, 0.95333333333333337, 0.96666666666666656, 0.96666666666666656, 0.96666666666666679, 0.96666666666666679, 0.96666666666666679, 0.96666666666666679, 0.97333333333333338, 0.96666666666666679, 0.96666666666666679, 0.97333333333333338, 0.98000000000000009, 0.97333333333333338, 0.97333333333333338, 0.97333333333333338, 0.97333333333333338, 0.98000000000000009, 0.97333333333333338, 0.98000000000000009, 0.96666666666666656, 0.96666666666666656, 0.97333333333333338, 0.95999999999999996, 0.96666666666666656, 0.95999999999999996, 0.96666666666666656, 0.95333333333333337, 0.95333333333333337]\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# plot the K values (x-axis) versus the 10-fold CV score (y-axis)\n",
      "plt.plot(k_range, scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "[<matplotlib.lines.Line2D at 0x18f45080>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEACAYAAAC+gnFaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmwVeWZ7/HvwwGUQRmCDHKOQBSQQQGJCCR2TicxIX0z\n2Omq2PSgGcqmKm3i7c6tckh1xE5Vt6bLdOi2KrE7dtrkxsS+fUM0lTZqpzy5fZtBkMMB5SDikMsk\nKjSDIjI99493LVhs9rzXnn+fqlPsvYZ3v4t1znr2+z7vu5a5OyIi0r4G1LsCIiJSXwoEIiJtToFA\nRKTNKRCIiLQ5BQIRkTanQCAi0uYKBgIzW2JmW83sRTO7Lcv6UWa20sz6zGytmc1KrLvDzJ43s81m\n9rCZnRctH21mT5nZNjN70sxGpntYIiJSrLyBwMw6gPuBJcBMYKmZzcjY7E5gg7vPAW4EVkT7TgZu\nBq5y9yuADuD3o31uB55y92nAr6L3IiJSB4VaBAuA7e7+qrsfB34CfDpjmxnA0wDu/gIw2cwuAg4B\nx4GhZjYQGArsivb5FPBQ9Poh4PpKD0RERMpTKBBMBHYk3u+MliX1AZ8BMLMFwCSg0933A/cB/w/Y\nDRx093+P9hnn7nuj13uBcWUfgYiIVKRQICjm/hP3ACPNrBe4BegFTprZpcB/ByYDFwPDzOwPz/mA\ncI8L3edCRKROBhZYvwvoSrzvIrQKTnP3w8AX4vdm9grwMvDfgFXuvi9a/lNgMfAjYK+ZjXf318xs\nAvB6tg83MwUIEZESubuVsn2hFsF6YKqZTTazwcANwGPJDcxsRLQOM7sZ+LW7vwW8ACw0syFmZsBH\ngC3Rbo8BN0WvbwJ+lueAWvLnrrvuqnsdWvn4fvGL0NBctqw1j+8v/zIc3z/8Q7rlPvdcKLe7u77H\nV+2fep+/av6UI28gcPcThO6eJ6KL+CPu3m9my8xsWbTZTGCzmW0FPgbcGu27EfgBIZhsirb9h+jf\ne4DrzGwb8KHovUhqVq2Cj340/NuKqnV8q1bBoEFw8GC65UpjK9Q1hLs/DjyeseyBxOvVwPQc+34T\n+GaW5fsJLQSRqli9Gr7yFfjDP4RDh+DCC+tdo/ScOgVr1sAjj4RjTNPq1fCxj0F/f7rlSmPTzOI6\n6e7urncVqqqex3fiBDzzDFx7LVx1Faxdm/5n1PP4tm6F0aPhwx+G3bth3770yl61Cj77WTh5sju9\nQhtQq//9lUqBoE5a/Rexnsf33HPQ2RkulosXh2+5aavn8a1eHY6rowMWLAitgzTs2xcCy5IlsH9/\ndzqFNqhW//srlQKBtJxVq2DRovB60aLWyxMkj2/x4vSOb82aEFjGjAmtqkOH0ilXGp8CgbScVavC\nBRLCBXPNmtCv3ioyjy+tQBCXawZdXbBjR+F9pDUoEEjLibtOAMaOhYsuap3k5/79sGsXzJ4d3i9c\nCOvXh2/wlUr+vykQtBcFAmkpe/eGi+Xll59Z1krdQ2vWwNVXw8BovN+oUeGivWlT/v0KOXEC1q2D\na64J77u6YOfO/PtI61AgkJayenX4ljwg8ZtdrYRxPSS/tcfSOL5Nm+CSS0JggZBsV4ugfSgQSEtJ\nJlJjrdQiyHZ8aSSMV68+u1x1DbUXBQJpKclEamz27PTH29dD3H2zcOHZy9MIdJn/bwoE7UWBQFrG\nsWPQ2xuGQCZ1dIS+77TG29dLcn5E0vTp4ZYQe/aUX3Zml5MCQXtRIJCW0dsLl12W/XYSrdA9lK1b\nCEI+ZNGi8vMEe/bAgQMwbdqZZXGyuMx7mEmTUSCQlpEtkRprhYRxvuOrJBDE+YFkgv3CC8P7AwfK\nK1OaiwKBtIxc35ghdA2tW5fOePt6yXd8lSSMMxPFMXUPtQ8FAmkZ2RLFsVGjwvDISsfb10u2+RFJ\nCxbAxo3w7rull53r/02BoH0oEEhL2LEjJIsvvTT3Ns3cPZSt+yZp+PDQx9/bW1q5774LfX3nJthB\ngaCdKBBIS4i7TSzPA/qaOWGcr1soVk73UG8vTJ0aAkkmzS5uHwoE0hLyJVJjzd4iKHR85SSM83Wn\naXZx+1AgkJZQzDfmadPCKJhKxtvXQ675EZniFkEpQz5zJYpBXUPtRIFAmt4778Dzz8P73pd/u0rH\n29dLPD/iggvybzdlCpw8WfzF2z1/i0CBoH0oEEjTW78eZs2CoUMLb9uM3UPFdAtByI+UkgfZsSME\njilTsq/XpLL2oUAgTa+YbqFYMyaMSzm+UhLGhRLsw4bBkCHNf48mKUyBQJpesd+YIfSz9/WVN96+\nXko5vlK6vvJ1C8WUMG4PCgTS1Ar1c2cqd7x9vcTzI9773uK2nz8ftmyBI0cKb5svURxTnqA9KBBI\nU3v5ZRg8OFywitVM3UPFzI9IGjIErrgi5E3yOXIkBIz58/Nvp0DQHhQIpKmV0hqINVPCuJRuoVgx\ngW79+hAwhgzJv50CQXtQIJCmVkoiNRZfKJthNEw5x1dMoCu2XM0ubg8KBNLUyvnGXOp4+3opdn5E\npmICXbEtKSWL24MCgTStw4dh+3aYN6+0/czSec5vta1fHx6zWaj7JlNnZ9hn+/bs692LSxSDuoba\nRcFAYGZLzGyrmb1oZrdlWT/KzFaaWZ+ZrTWzWdHy6WbWm/g5aGZfidYtN7OdiXVL0j80aXXPPANz\n54ZkcamaIWFcTrdQLF/30PbtIVB0dhYup7MTdu2CU6fKq4c0h7yBwMw6gPuBJcBMYKmZzcjY7E5g\ng7vPAW4EVgC4+wvuPs/d5wHzgSPAymgfB74Vr3f3X6Z2RNI2ykkUx5ohYVxOt1csX6Ar5f9tyJDw\ntLLXXy+vHtIcCrUIFgDb3f1Vdz8O/AT4dMY2M4CnIVz8gclmdlHGNh8BXnL3ZCOzyAFxItlV8o25\nlPH29RDPj6hGi6DYbqGYEsatr1AgmAgkL947o2VJfcBnAMxsATAJyGx0/j7wcMayL0fdSQ+a2ciS\nai1t79QpWLOm/Avl+ecXN96+Xl5+Gc47r7T5EUlz58JLL8GhQ+euK7UlpYRx6ysUCIoZYHcPMNLM\neoFbgF7gZLzSzAYDnwT+V2Kf7wBTgLnAHuC+EuoswtatMHo0jB9ffhmNnDCupNsLYNAguOoqWLv2\n7OWHDoUgM2dO8WUpYdz6BhZYvwtIfifpIrQKTnP3w8AX4vdm9grwcmKTjwPPuvsbiX1eT2z/PeDn\nuSqwfPny06+7u7vp7u4uUGVpB6V2b2SzaBH88Ifp1CdtlXQLxeLuoeuuO7Ns7doQIEpJsCsQNLae\nnh56enoqKqNQIFgPTDWzycBu4AZgaXIDMxsBvOPux8zsZuDX7v5WYpOlwI8z9png7vHjQX4X2Jyr\nAslAIBKr9BszhAvtl74U+uOLvYVDraxeDZ//fGVlLFoE3/nO2cvK+X/r6oKNGyuri1RP5hfku+++\nu+Qy8nYNufsJQnfPE8AW4BF37zezZWa2LNpsJrDZzLYCHwNujfc3s2GERPFPM4q+18w2mVkf8EHg\nz0quubS1NFoEhcbb10s8P2Lu3MrKWbQotACSQz/L+X9Tsrj1FWoR4O6PA49nLHsg8Xo1MD3Hvm8D\nY7Isv7HkmopE9u8PXRVXXFF5WXH3ydSplZeVlmeeCZPkypkfkTR2LIwZA/394cE9cYL9Bz8orRwl\ni1ufZhZL01mzJjxXYGDBrzGFNWLCOI1ur1jy+QT9/XDRRSFAlGLixPCc55MnC28rzUmBQJpOGt1C\nsUacYZxGojiWDHTllnveeWGE1muvpVMnaTwKBNJ00vzGPHduGE6Zbbx9PVQ6PyJTZiAo9/9NI4da\nmwKBNJUTJ2DdOli4MJ3yco23r5d4fsS4cemUN2tW6NbZt6+ylpQSxq1NgUCaynPPhT7r0aPTK7OR\n7jtUyf2FsunoCPmUX/wCdu8OdzMthxLGrU2BQJpKmt1CsUZKGFfj+BYtgr/9W7jmmhAYyqGuodam\nQCBNJc1EcWzhwtAv3wi3Wk4zURxbvDhMCKukXAWC1pbCADxpB2++GS6UpQ49TNuqVXDHHemWOXZs\nGFb5ve/BhAnpll2Kd98N9/4vt/smlzifUklLo9aB4Phx+M1v4LLLaveZ7UyBQIryjW+EC9V3v1u/\nOuzdGyaTXX55+mXfeiv8POcdr2rnq19NZ35E0siR8LWvwfvfX34ZtU4WP/oo/M3fNE4Sv9WZN/AT\nvM3MG7l+7eTqq0Mg2LSpfnX42c/ggQfg8ccLbyvpOn4chg0Lz29IO1Bl8+d/Dj/+cRjxJKUxM9y9\npLtnKUcgBR05Eh7i8vLLcPBg/epRjUSqFGfQoNB9tnt3bT5v1arQAjx2rDaf1+4UCKSg+CHqV10V\n7oNTL9VIFEvxapUnOHoUNm8O90natav6nycKBFKE+Jt4PYdZHjsGvb1hTLzUR60CwbPPwowZ4UaA\nmsRWGwoEUlD8TTx5A7Na27gRLr00PEhd6qNWCeP4901DVmtHgUDyih+ivnhx+OOs13h75Qfqr1az\ni+NzrUBQOwoEktf27eHhLZ2dZ8bb9/fXvh4KBPVXiwuzu1oE9aBAIHllJmjrddtmJYrrrxYX5ldf\nDY8NnTRJ9zeqJQUCySvzm3g9EsY7doQ5DJdeWtvPlbPVIhDEv29muuNpLSkQSF7ZWgS1ThjHdWi0\nB8y3m/Hjw8zuao7tT/6+qWuodhQIJKdDh+Cll85+iPrs2WFS0b59tauH8gONoaMjBINqju1Pnuux\nY8Pv4NGj1fs8CRQIJKe1a8MksuRD1Ds6wu2M16ypXT0UCBpHNb+lv/UWvPBC+J0DGDAALr5Y3UO1\noEAgOeVK0NYyYfzOO/D88/C+99Xm8yS/agaCdetgzpzwjOSYEsa1oUAgOeX6Jl7LhPH69eFxi0OG\n1ObzJL9qBoJsv29KGNeGAoFkdepU6BrK1iK45ppwgT5xovr10LDRxlLNC3O2c62EcW0oEEhW/f3w\nnvdkfxDNqFFwySW1uSW18gONpVpdNcmJZEkKBLWhQCBZFboA1+KB78nbW0hjqNaFeds2uOCCkByu\nxefJ2RQIJKtCXTK1SBi//HIYsdTVVd3PkeJV68KcK+ArWVwbCgSSVTEtgmoHArUGGk88tv+dd9It\nN9e5VrK4NhQI5Bz79oVJY/keoj5tWnhaWTUfJbhqlRLFjWbAAJg4Mf1JZblaoGPGhCfkHTmS7ufJ\n2QoGAjNbYmZbzexFM7sty/pRZrbSzPrMbK2ZzYqWTzez3sTPQTP7SrRutJk9ZWbbzOxJMxuZ/qFJ\nudasCQ+A6ejIvc2AAdW/3cTq1WoRNKK0u2sOHIDf/AauvPLcdWbqHqqFvIHAzDqA+4ElwExgqZnN\nyNjsTmCDu88BbgRWALj7C+4+z93nAfOBI8DKaJ/bgafcfRrwq+i9NIhiu2SqmTA+fDjcAnvevOqU\nL+VLO0+wdi3Mnx+ei1yLz5NzFWoRLAC2u/ur7n4c+Anw6YxtZgBPQ7j4A5PN7KKMbT4CvOTu8en8\nFPBQ9Poh4Poy6y9VUOzY/WomjJ95JtzjKHl7C2kMaV+YC33xUIug+goFgolA8hTsjJYl9QGfATCz\nBcAkoDNjm98HHk68H+fue6PXe4FxJdRZqujEiTDVf+HCwtsuWAB9feEW0WlTorhxpR0ICnUBKmFc\nfQMLrPciyrgHWGFmvcBmoBc4Ga80s8HAJ4Fz8gsA7u5mlvNzli9ffvp1d3c33d3dRVRJyrVpU/jD\nGzWq8LbDh4cHjPf2Fhc4SrFqFfzJn6RbpqSjqwsefzydsk6eDF1D+X5/urpgw4Z0Pq8V9fT00NPT\nU1EZhQLBLiA5iruL0Co4zd0PA1+I35vZK8DLiU0+Djzr7m8klu01s/Hu/pqZTQBez1WBZCCQ6is1\nQRsPI00zEJw6FRLW3/9+emVKetLsqnn++XBr6zFjcm/T1QWPPprO57WizC/Id999d8llFOoaWg9M\nNbPJ0Tf7G4DHkhuY2YhoHWZ2M/Brd38rsclS4McZ5T4G3BS9vgn4Wck1l6ootUumGgnjrVtDi2T8\n+HTLlXSk2TVUTD5KyeLqyxsI3P0EcAvwBLAFeMTd+81smZktizabCWw2s63Ax4Bb4/3NbBghUfzT\njKLvAa4zs23Ah6L30gBKvclbnDD2YjoRS6iD8gONa8yYMKHs7bcrL6uYLx5KFlefeZp/wSkzM2/k\n+rWaPXvCLZ/ffDPMEyiGO0yYEEb5XHJJOvX44hfDcMIvfSmd8iR9U6fCz38Ol19eWTnTpsFPf5p/\n8qJ7yEft2QMXXljZ57UDM8PdS3qwq2YWy2mrV4e+/mKDAIQJP2kPI9WM4saXxkieN96A11+HmTPz\nb6cH2VefAoGcVm6XTJr3Hdq/P/zBX3FFOuVJdaTRXbN6dXi2RTFfPJQnqC4FAjmt3LH7aSaM16yB\nq6+GgYXGs0ldpXFhLiUfpUBQXQoEAoRJYRs3hklipZo/H7ZsSefGYEoUN4c0LsylfPFQwri6FAgE\nCJPCpk0LSblSnX9+6MpZv77yemhGcXOoNBAcPx4miV1zTfGfpxxB9SgQCFB5gjaNhPGJE2H0Udqz\nlCV9lV6Y+/pg8mQYMaL4z1OLoHoUCASovEsmjYTxc8+FLoDRoysrR6qv0q6aUlt+CgTVpUAgqTwb\nOH42QSXTPjRstHmMGhVacIcOlbd/qRMX40CgaUXVoUAg7NgR/qinTCm/jM5OGDIkPEOgXEoUN494\nbH+539JL/eJx4YVhmOmBA+V9nuSnQCCn/yitpLmI56p0GKkSxc2l3ECwa1e4PcXUqaV/nhLG1aFA\nIKl1yVSSMN67N0wmq/SWBVI75V6Y426hUr94KE9QPQoEklqXTCUtgnJubyH1VW7CuNzfNwWC6tGf\nXZt7++0wGWz+/MrLmjsXXnqpvASiEsXNp9wLc7nnWoGgehQI2tz69eHOj0OGVF7WoEFw1VXhiVOl\nUn6g+ZRzYT56NDwF7+qrS/88zS6uHgWCNpf2SJ1yuoeOHQszm8u5vYXUTzmBYMMGmDEDhg0r7/OU\nLK4OBYI2l3aXTDkJ495euOwy3Wu+2cQX5lLG9lfy+6auoepRIGhj7um3CBYtCl1Dp04Vv4/mDzSn\ncsb2V3Kuywk8UhwFgja2fXu4YVxnZ3pljh0L73kP9PcXv48Sxc2rlG/p8Qz2cs/1sGHh93XfvvL2\nl9wUCNpYtRK0pd53SIni5lVKIHj11TB3YNKk8j9PCePq0OM/8ujpgfvuq3ctqmfrVrjllvTLXbwY\n7rkHHnus8LYnT4Zk8aWXpl8Pqb5LLoGvfQ2++93C277xRuUz2OPuoXnzyi/j9tvhrrvSGSmX9PDD\nYbZ0OSOi6k2BII9f/jJ0c/ze79W7JtXT3Z1+mX/8x+GbW7F9uZ2dld/eQurj618Pyf5izZlT2edV\nmjDevRvuvRd+53fgt36rsrpk+vu/h09+UoGg5ezYAUuWhJMrxRs2DD7xiXrXQmqhszPdHFMhlQaC\neGjzqlXpBoKjR+HZZ8OkymakHEEeO3fW9pdcRPJLIxDMnZveM7ZjGzaEp6416zwHBYI8duwIv3gi\n0hjSeCDOV78a/k1zGOqqVfD+9zdvIluBIIdTp8LtctUiEGkclcwufvfd8IjM66+v/NkZmVatgs9+\nVoGg5bz+enie6vnn17smIhLr7Axf0EqZsBjbsAGmT4fhw888US8N8cTMT34SjhwJP81GgSAHdQuJ\nNJ4hQ+CCC8JQ1FIl56uk8YztWDw/YvJkmDixOVsFCgQ5KFEs0pjKTRgnb29R6dP0spUbP76zGRPG\nBQOBmS0xs61m9qKZ3ZZl/SgzW2lmfWa21sxmJdaNNLN/NbN+M9tiZtdEy5eb2U4z641+lqR7WJVT\ni0CkMZWTMHaH//zPM7e3mDOn/GdnZEreNqNZb4yXNxCYWQdwP7AEmAksNbMZGZvdCWxw9znAjcCK\nxLoVwL+5+wzgSmBrtNyBb7n7vOjnl5UfSroUCEQaUznfun/zm/Dv5Mnh38GDy392RqZkl1NLBgJg\nAbDd3V919+PAT4BPZ2wzA3gawN1fACab2UVmNgK41t3/KVp3wt0PJvZr6LmkCgQijamci2225ySn\nkTB+6y144YUQVMqtWyMoFAgmAsnD2hktS+oDPgNgZguASUAnMAV4w8y+b2YbzOwfzWxoYr8vR91J\nD5rZyIqOogoUCEQaUzkX22w3NkwjYbxuXehmOu+88L6zszlzBIVuMVHMlIt7gBVm1gtsBnqBk8Bg\n4CrgFndfZ2bfBm4Hvg58B/jLaP9vAPcBX8xW+PLly0+/7u7uprsaN8fJQslikcZUbiBYuvTsZYsW\nwec+F4aiDihz2Ezm8xXq0SLo6emhp6enojLM80yvM7OFwHJ3XxK9vwM45e735tnnFeAKYDiw2t2n\nRMs/ANzu7p/I2H4y8HN3vyJLWZ6vftVy8iQMHQqHD4e+RBFpHC+/DL/922f6/Qt5++3wnIx9+86d\nF3TZZfDoozBrVvZ9C/nEJ+Dznz9zY8r9++G97y3tYT1pMzPcvaSu90JxcD0w1cwmm9lg4AbgrJsL\nm9mIaB1mdjPwa3d/y91fA3aY2bRo048Az0fbTUgU8buElkTDeO01GD1aQUCkEU2cCHv2hC9sxVi3\nDq68Mvvk0Eq6h+KJZMkH7YwaFe45dPhweWXWS95A4O4ngFuAJ4AtwCPu3m9my8xsWbTZTGCzmW0F\nPgbcmijiy8CPzKyPMGror6Ll95rZpmj5B4E/S+2IUqD8gEjjOu+88EVt797its+8WCdVkjDeti1M\nbrv44jPL4rkEzZYwLngband/HHg8Y9kDidergek59u0Dzrk7t7vfWHJNa0iBQKSxxRfb5EU4l1Wr\n4Kabsq9bvBhWrMi+rphysz1ZL04Yz5xZXrn1oJnFWShRLNLYiv3WHXff5HoU6uzZoZupnOcg5yq3\nGVsECgRZqEUg0tiKvdi++GJ4UFKulkNHR3ii2Jo1pdchOaO4nLo1EgWCLBQIRBpbseP1c3XfJJWT\nMD5wIIxauvLKc9cpELQIBQKRxlbsxTZfojhWTsJ47VqYPx8GDSq/bo1EgSALBQKRxlbsxbaYFsHC\nhWGI6YkTxX9+vnKbcXaxAkGG48fDvc4nTCi8rYjURzGB4OBBeOWVcAuIfEaNgksugc0lzGbKl4CO\n61aHubBlUyDIsGdPmIU4sODAWhGplwkTwlME832Lz9d9k2nRouLzBCdPhrIXLsy+fsSIMJ/g4MHs\n6xuRAkEGdQuJNL5Bg+Cii8IXt1yK6RaKlZIwfv55GD8exozJvU2z5QkUCDIoEIg0h0IX22ISxbFS\nEsbFlKtA0OQUCESaQ76L7cmTYW5AsYFg+vQwJDRfCyNWTEuj2RLGCgQZNKtYpDnkCwRbtsC4caH7\nqBgDBhTfKigmEKhF0OTUIhBpDvkutqV0C8WKCQRvvBF+Ct1HSIGgySkQiDSHfN0vpSSKY8UkjFev\nhmuuKfwgGwWCJqdAINIc0m4RLFgAGzfCu+/m3qbYchUImti774YnDI0bV++aiEghuS62b74ZHi5V\n6lPHhg+HadOgtzf3NsW2NOLWSrNMKlMgSNi9O0xU6eiod01EpJDx48MXt2PHzl4ed9+U83ecr3vo\n+HF49tlQdiHDh4cH6OzfX3od6kGBIEHdQiLNo6MjBINdu85eXk63UCxfwrivD6ZMCTOHi9FM3UMK\nBAkKBCLNJVvCuJxEcSxuEWTr0im1XAWCJqVAINJcMi+2pXTfZDNlSrh/UbYLeKktDQWCJqVAINJc\nMi+2mzbBpEkwcmR55ZnlzhOU2iJoptnFCgQJmlUs0lwyA0El3UKxbIFg1y54+22YOrX8ujUyBYIE\ntQhEmkvmxbaSRHEsW8I4Ltes/Lo1MgWCBAUCkeaS2f2SRotg/vxwr6IjRyorV4GgCb3zDhw+XPxN\nqkSk/pIX2927w9/wtGmVlTlkCMyeDevXn1lWTkujszN0KZ06VVl9akGBILJzJ0ycWPgeIiLSOMaO\nhUOH4OjR8rpvcknmCY4eDUnoq68urYwhQ8LEsjffrLw+1abLXkSJYpHmM2AAXHxx+PtNo1solgwE\nzz4LM2bAsGGll9Ms3UMKBBHlB0SaU3yxTSNRHIsTxu6VlatA0GQUCESaU2cnvPRSuAVEqd03+coc\nMgS2b6+spdEygcDMlpjZVjN70cxuy7J+lJmtNLM+M1trZrMS60aa2b+aWb+ZbTGzhdHy0Wb2lJlt\nM7MnzazM6R/pUSAQaU5dXfDoo+Fxk8OHp1fuokUhCLR9i8DMOoD7gSXATGCpmc3I2OxOYIO7zwFu\nBFYk1q0A/s3dZwBXAv3R8tuBp9x9GvCr6H1dKRCINKeuLnjiifTyA7HFi+Hhh0PyedKk8spoltnF\nhVoEC4Dt7v6qux8HfgJ8OmObGcDTAO7+AjDZzC4ysxHAte7+T9G6E+5+MNrnU8BD0euHgOsrP5TK\nKFks0py6usI9hqoRCJ58Mvxb7kiklmgRABOB5GHsjJYl9QGfATCzBcAkoBOYArxhZt83sw1m9o9m\nNjTaZ5y7741e7wUqehTM/v35nypUDLUIRJpT/HebVqI4NmcOnH9+ZeU2SyAYWGB9Mc/XuQdYYWa9\nwGagFzgJDAauAm5x93Vm9m1CF9DXz/oAdzeznJ+zfPny06+7u7vp7u4+Z5vrr4e/+Au47roiapvF\n22+HscLveU95+4tI/bz3vfDRj8LkyemWO3gw3HBDKLtcEyfCnj1w8mT1HnjV09NDT09PRWWY53mW\nWpTcXe7uS6L3dwCn3P3ePPu8AlwBDAdWu/uUaPm1wG3u/gkz2wp0u/trZjYBeNrdL89SluerX+y2\n22DoULjrroKbZrV1K3zqU7BtW3n7i4jkMn48bNgQ5jvUgpnh7iV1ZhXqGloPTDWzyWY2GLgBeCzj\nQ0dE6zCzm4Ffu/tb7v4asMPM4gnfHwaej14/BtwUvb4J+Fkplc60eHHupwoVQ91CIlItzZAwzhsI\n3P0EcAvwBLAFeMTd+81smZktizabCWyOvuV/DLg1UcSXgR+ZWR9h1NBfRcvvAa4zs23Ah6L3ZVu0\nCNasKf93bOg8AAAK4klEQVSeHkoUi0i1NEOeoFCOAHd/HHg8Y9kDidergek59u0Dzpni4e77gY+U\nWtlcxo6FMWOgvx9mzSq8fSa1CESkWpohELTMzOJcTxUqhgKBiFSLAkENxbMAy6FAICLVokBQQ5Uk\njBUIRKRamj5Z3Exmzw4Ppti3r/R9lSwWkWpRi6CGOjpgwYIweqgUhw6FyR4j637bOxFpRRdfDHv3\nwokT9a5Jbi0TCKC8hHHcLZTGU41ERDINGhQegbtnT71rkltLBYJyEsbKD4hItTV691BLBYKFC8MD\np0tpgikQiEi1NXrCuKUCwahRcMkl4UHTxVKiWESqTS2CGoufNVostQhEpNoUCGqs1ISxAoGIVJsC\nQY2pRSAijUaBoMamT4cDB4obquWuQCAi1adkcY0NGFB8q+DAARg4EC64oPr1EpH2NWECvPkmHDtW\n75pk13KBAIoPBGoNiEgtdHSEJ5Xt3l3vmmTXkoGg2ISxAoGI1Eoj5wlaMhAsWAAbN8K77+bfToFA\nRGpFgaDGhg+HadOgtzf/dgoEIlIrjZwwbslAAMV1D2lWsYjUiloEdVBMwlgtAhGpFQWCOohbBO65\nt1EgEJFaUSCogylTwl1Ic/3Hu4euIQUCEakFBYI6MMufJ3jzTRg6NPyIiFTb2LFw8CAcPVrvmpyr\nZQMB5A8EShSLSC0NGBAeW7lrV71rcq6WDgT5EsbKD4hIrTVq91BLB4L582HLFjhy5Nx1CgQiUmsK\nBHUwZAjMnh0eX5lJgUBEak2BoE5y5QkUCESk1rq6GnN2ccFAYGZLzGyrmb1oZrdlWT/KzFaaWZ+Z\nrTWzWYl1r5rZJjPrNbNnEsuXm9nOaHmvmS1J75DOlisQKFksIrXW2dmELQIz6wDuB5YAM4GlZjYj\nY7M7gQ3uPge4EViRWOdAt7vPc/cFGcu/FS2f5+6/rPRAcokTxpkTy9QiEJFaa9auoQXAdnd/1d2P\nAz8BPp2xzQzgaQB3fwGYbGYXJdZbjrJzLU9VZ2fIFWzffmbZqVNhCJdaBCJSS80aCCYCyWrvjJYl\n9QGfATCzBcAkIL7EOvDvZrbezG7O2O/LUXfSg2Y2sqzaFylzGOnrr8OIEXD++dX8VBGRs40ZE0Yx\nZhvJWE+FAkGeO/Wcdg8w0sx6gVuAXuBktO4D7j4P+Djwp2Z2bbT8O8AUYC6wB7iv1IqXIjNPoG4h\nEakHs8a8HfXAAut3AclLZhehVXCaux8GvhC/N7NXgJejdbujf98ws5WErqb/cPfXE9t/D/h5rgos\nX7789Ovu7m66u7sLVPlcixfDgw+eea9EsYjUS5wwnjYtnfJ6enro6empqAzzPLfnNLOBwAvAh4Hd\nwDPAUnfvT2wzAnjH3Y9F3T/vd/fPmdlQoMPdD5vZMOBJ4G53f9LMJrj7nmj/PwOudvc/yPL5nq9+\nxTp2DEaPDs8LvfBC+Lu/g23b4P77Ky5aRKQkN94IH/oQfO5z1SnfzHD3knKweVsE7n7CzG4BngA6\ngAfdvd/MlkXrHyCMJvpnM3PgOeCL0e7jgJVmFn/Oj9z9yWjdvWY2l9D19AqwrJRKl2rwYLjqKli7\nFq67Tl1DIlI/jZgwLtQ1hLs/DjyeseyBxOvVwPQs+71CyAFkK/PGkmtaoThhHAeCefNqXQMRkRAI\nNmyody3O1vIzi2PJhLFaBCJSL404u7htAsGiRbBmTZhDoGSxiNRLI84ubptAMHZsGMP73HPw2msw\nMXM2hIhIDTRijqBtAgGE7qGVK8MIosGD610bEWlHo0bB8eNw+HC9a3JGWwWCRYvgkUeUHxCR+jFr\nvFZBWwWCxYuhv1+BQETqq9ESxm0VCGbPhuHDlSgWkfpqtIRxwXkEraSjA665Ri0CEamvri54+OHi\ng8Fll8Ef/VH16tNWgQDgG9+ACRPqXQsRaWd/8AcwoIH6Y/Lea6je0rrXkIhIuyjnXkMNFJNERKQe\nFAhERNqcAoGISJtTIBARaXMKBCIibU6BQESkzSkQiIi0OQUCEZE2p0AgItLmFAhERNqcAoGISJtT\nIBARaXMKBCIibU6BQESkzSkQiIi0OQUCEZE2p0AgItLmFAhERNpcwUBgZkvMbKuZvWhmt2VZP8rM\nVppZn5mtNbNZiXWvmtkmM+s1s2cSy0eb2VNmts3MnjSzkekdkoiIlCJvIDCzDuB+YAkwE1hqZjMy\nNrsT2ODuc4AbgRWJdQ50u/s8d1+QWH478JS7TwN+Fb1vKz09PfWuQlXp+Jqbjq+9FGoRLAC2u/ur\n7n4c+Anw6YxtZgBPA7j7C8BkM7sosT7bQ5Q/BTwUvX4IuL7Uije7Vv9F1PE1Nx1feykUCCYCOxLv\nd0bLkvqAzwCY2QJgEtAZrXPg381svZndnNhnnLvvjV7vBcaVUXcREUnBwALrvYgy7gFWmFkvsBno\nBU5G6z7g7rujFsJTZrbV3f/jrA9wdzMr5nNERKQKzD33NdjMFgLL3X1J9P4O4JS735tnn1eAK9z9\nrYzldwGH3f1bZraVkDt4zcwmAE+7++VZylKAEBEpkbtn65LPqVCLYD0w1cwmA7uBG4ClyQ3MbATw\njrsfi7p/fu3ub5nZUKDD3Q+b2TDgo8Dd0W6PATcB90b//iyNgxERkdLlDQTufsLMbgGeADqAB929\n38yWResfIIwm+ufo2/tzwBej3ccBK80s/pwfufuT0bp7gH8xsy8CrwKfTfWoRESkaHm7hkREpPU1\n5MziQpPYml2uiXbNysz+ycz2mtnmxLKWmTSY4/iWm9nO6Bz2mtmSetaxXGbWZWZPm9nzZvacmX0l\nWt4S5y/P8bXK+Ts/msi70cy2mNlfR8tLOn8N1yKIJrG9AHwE2AWsA5a6e39dK5aiKKE+393317su\naTCza4G3gB+4+xXRsm8Cb7r7N6NgPsrdm3LiYI7jOz34oa6Vq5CZjQfGu/tGMxsOPEuY1/N5WuD8\n5Tm+z9IC5w/AzIa6+xEzGwj8X+B/EOZqFX3+GrFFUMwktlbQMonwaEjwf2UsbplJgzmOD1rgHLr7\na+6+MXr9FtBPmCvUEucvz/FBC5w/AHc/Er0cTMjl/hclnr9GDATFTGJrdrkm2rWSdpg0+OXoHlsP\nNmvXSVI0OnAesJYWPH+J41sTLWqJ82dmA8xsI+E8Pe3uz1Pi+WvEQNBYfVXV8X53nwd8HPjTqOuh\nZXnof2y18/odYAowF9gD3Fff6lQm6jb538Ct7n44ua4Vzl90fP9KOL63aKHz5+6n3H0u4Y4Ov2Vm\nv52xvuD5a8RAsAvoSrzvIrQKWoa774n+fQNYSegOazV7o/5ZokmDr9e5Pqly99c9AnyPJj6HZjaI\nEAR+6O7xnJ6WOX+J4/uf8fG10vmLuftB4BfAfEo8f40YCE5PYjOzwYRJbI/VuU6pMbOhZnZB9Dqe\naLc5/15NKZ40CHkmDTar6I8r9rs06Tm0MNHnQWCLu387saolzl+u42uh8zcm7tYysyHAdYTb/JR0\n/hpu1BCAmX0c+DZnJrH9dZ2rlBozm0JoBcCZiXZNfXxm9mPgg8AYQn/k14FHgX8BLiGaNOjuB+pV\nx0pkOb67gG5Ct4IDrwDLEn2yTcPMPgD8H2ATZ7oP7gCeoQXOX47ju5Nwh4RWOH9XEJLBA6KfH7r7\n35jZaEo4fw0ZCEREpHYasWtIRERqSIFARKTNKRCIiLQ5BQIRkTanQCAi0uYUCERE2pwCgYhIm1Mg\nEBFpc/8fRCKt63G6jx4AAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x219ab00>"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Default dataset\n",
      "\n",
      "- Binary classification\n",
      "- Using AUC as evaluation metric\n",
      "- Using cross-validation to select between models"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# read in the default data\n",
      "data = pd.read_csv('https://raw.githubusercontent.com/justmarkham/DAT5/master/data/default.csv')\n",
      "X = data[['balance']]\n",
      "y = data.default"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# 10-fold cross-validation with logistic regression\n",
      "logreg = LogisticRegression()\n",
      "cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "0.94819099843061938"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# 10-fold cross-validation with KNN (K=5)\n",
      "knn = KNeighborsClassifier(n_neighbors=5)\n",
      "cross_val_score(knn, X, y, cv=10, scoring='roc_auc').mean()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "0.81417567491604037"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Advertising dataset\n",
      "\n",
      "- Regression\n",
      "- Using RMSE as evaluation metric\n",
      "- Using cross-validation to select features"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# read in the advertising data\n",
      "data = pd.read_csv('http://www-bcf.usc.edu/~gareth/ISL/Advertising.csv', index_col=0)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# 10-fold cross-validation with three features\n",
      "X = data[['TV', 'Radio', 'Newspaper']]\n",
      "y = data.Sales\n",
      "lm = LinearRegression()\n",
      "scores = cross_val_score(lm, X, y, cv=10, scoring='mean_squared_error')\n",
      "print scores"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[-3.56038438 -3.29767522 -2.08943356 -2.82474283 -1.3027754  -1.74163618\n",
        " -8.17338214 -2.11409746 -3.04273109 -2.45281793]\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# convert from MSE to RMSE\n",
      "scores_sqrt = np.sqrt(-scores)\n",
      "print scores_sqrt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[ 1.88689808  1.81595022  1.44548731  1.68069713  1.14139187  1.31971064\n",
        "  2.85891276  1.45399362  1.7443426   1.56614748]\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# calculate the average RMSE\n",
      "print np.mean(scores_sqrt)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1.69135317081\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# 10-fold cross-validation with two features\n",
      "feature_cols = ['TV', 'Radio']\n",
      "X = data[feature_cols]\n",
      "print np.mean(np.sqrt(-cross_val_score(lm, X, y, cv=10, scoring='mean_squared_error')))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1.67967484191\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Improvements to Cross-Validation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Although standard K-fold cross-validation is a very useful model evaluation procedure, there are some common variations that can make cross-validation even better:\n",
      "\n",
      "### Repeated cross-validation\n",
      "\n",
      "K-fold cross-validation is repeated multiple times (with different random splits of the data into the K folds), and the results are averaged. This provides a more reliable estimate of out-of-sample accuracy by reducing the variance associated with a single trial of cross-validation.\n",
      "\n",
      "### Creating a hold-out set\n",
      "\n",
      "Instead of running cross-validation on the entire dataset, a portion of the data is \"held out\" and not touched during the model building process. The best model is located and tuned using cross-validation on the remaining data. At the end of this process, the hold-out set is then used to test the best model. The hold-out set accuracy is considered to be a more reliable estimate of out-of-sample accuracy than the cross-validated accuracy, since the hold-out set is truly out-of-sample data.\n",
      "\n",
      "### Feature engineering and selection within cross-validation iterations\n",
      "\n",
      "Ideally, all feature engineering and selection should be done within each cross-validation iteration. Performing these tasks before cross-validation does not properly mimic the application of the model to out-of-sample data, since those processes will have \"unfair\" knowledge of the entire dataset, and thus the cross-validated estimate of out-of-sample accuracy will be biased upward. Performing these tasks within cross-validation iterations will produce a more reliable estimate."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}