Chapter 06 - Ensemble Methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.1\n",
    "\n",
    "Why is bagging based on random sampling with replacement? Would bagging still reduce a forecast's variance if sampling were without replacement?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: So every training set winds up with a disparate set of data, yes this is a technique called \"pasting\".**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.2a\n",
    "\n",
    "Suppose that your training set is based on highly overlap labels (i.e. with low uniqueness, as defined by Chapter 4)\n",
    "\n",
    "Does this make bagging prone to overfitting, or just ineffective? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Bagging helps with overfitting, but the selected training sets will likely still be very similar, thus not helping improve accuracy much.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.2b\n",
    "\n",
    "Suppose that your training set is based on highly overlap labels (i.e. with low uniqueness, as defined by Chapter 4)\n",
    "\n",
    "Is out-of-bag accuracy generally reliable in financial applications? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: High levels of serial correlation in financial datasets mean that the further apart your training from your validation data the better -- which is why K-Fold CV is generally preferable.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.3a\n",
    "\n",
    "Build an ensemble of estimators, where the base estimator is a decision tree.\n",
    "\n",
    "How is this ensemble different from a RF?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OOB score: 0.92\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "\n",
    "\n",
    "X, y  = make_classification(random_state=1)\n",
    "bc = BaggingClassifier(DecisionTreeClassifier(), oob_score=True, random_state=1)\n",
    "bc.fit(X, y)\n",
    "print(\"OOB score:\", bc.oob_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: They are indeed very similar, however the RF uses a random subset of features to generate splits.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.3b\n",
    "\n",
    "Build an ensemble of estimators, where the base estimator is a decision tree.\n",
    "\n",
    "Using sklearn, produce a bagging classifier that behaves like an RF. What parameters did you have to set up, and how?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OOB score: 0.95\n"
     ]
    }
   ],
   "source": [
    "X, y  = make_classification(random_state=1)\n",
    "bc = BaggingClassifier(DecisionTreeClassifier(splitter='random'), oob_score=True, random_state=1)\n",
    "bc.fit(X, y)\n",
    "print(\"OOB score:\", bc.oob_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.4a\n",
    "\n",
    "Consider the relation between an RF, the number of trees it is composed of, and the number of features utilized:\n",
    "\n",
    "Could you envision a relation between the minimum number of trees needed in an RF and the number of features utilized?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: If the number of trees is much smaller than the number of features, there might be features that are ignored via the random sub-sampling.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.4b\n",
    "\n",
    "Consider the relation between an RF, the number of trees it is composed of, and the number of features utilized:\n",
    "\n",
    "Could the number of trees be too small for the number of features used?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: See above.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.4c\n",
    "\n",
    "Consider the relation between an RF, the number of trees it is composed of, and the number of features utilized:\n",
    "\n",
    "Could the number of trees be too high for the number of observations available?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A: Generally much too high numbers of decision trees will only materially increase processing time rather than accuracy.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6.5\n",
    "\n",
    "Consider the relation between an RF, the number of trees it is composed of, and the number of features utilized:\n",
    "\n",
    "How is out-of-bag accuracy different from statified k-fold (with shuffling) cross-validation accuracy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A:Same answer as above: High levels of serial correlation in financial datasets mean that the further apart your training from your validation data the better -- which is why K-Fold CV is generally preferable.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}