C2_W4_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**\n",
    "\n",
    "Please use the `Submit Assignment` button on the top (marked as 1) to submit your assignment for grading. ***Do not*** use the `Submit Assignment` button displayed on the right (marked as 2).\n",
    "\n",
    "If you do not see the `Submit Assignment` button marked as 1, save your assignment and then re-open it.\n",
    "\n",
    "<img src=\"images/submit_button.png\" style=\"padding-top: 5px;width: 60%;left: 0px;margin-left: 150px;margin-right: 0px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EZqu9a_ChWYv"
   },
   "source": [
    "# Cox Proportional Hazards and Random Survival Forests\n",
    "\n",
    "Welcome to the final assignment in Course 2! In this assignment you'll develop risk models using survival data and a combination of linear and non-linear techniques. We'll be using a dataset with survival data of patients with Primary Biliary Cirrhosis (pbc). PBC is a progressive disease of the liver caused by a buildup of bile within the liver (cholestasis) that results in damage to the small bile ducts that drain bile from the liver. Our goal will be to understand the effects of different factors on the survival times of the patients. Along the way you'll learn about the following topics: \n",
    "\n",
    "- Cox Proportional Hazards\n",
    "  - Data Preprocessing for Cox Models.\n",
    "- Random Survival Forests\n",
    "  - Permutation Methods for Interpretation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- [1. Import Packages](#1)\n",
    "- [2. Load the Dataset](#2)\n",
    "- [3. Explore the Dataset](#3)\n",
    "- [4. Cox Proportional Hazards](#4)\n",
    "    - [Exercise 1 - to_one_hot](#ex-1)\n",
    "- [5. Fitting and Interpreting a Cox Model](#5)\n",
    "- [6. Hazard ratio](#3)\n",
    "    - [Exercise 2 - hazard_ratio](#ex-2)\n",
    "- [7. Harrell's C-Index](#7)\n",
    "    - [Exercise 3 - harrell_c](#ex-3)\n",
    "- [8. Random Survival Forests](#8)\n",
    "- [9. Permutation Method for Interpretation](#9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IH0ukiNS3zG-"
   },
   "source": [
    "<a name='1'></a>\n",
    "## 1. Import Packages\n",
    "\n",
    "We'll first import all the packages that we need for this assignment. \n",
    "\n",
    "- `sklearn` is one of the most popular machine learning libraries.\n",
    "- `numpy` is the fundamental package for scientific computing in python.\n",
    "- `pandas` is what we'll use to manipulate our data.\n",
    "- `matplotlib` is a plotting library.\n",
    "- `lifelines` is an open-source survival analysis library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0JHzRJaQi_nU"
   },
   "outputs": [],
   "source": [
    "import sklearn\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from lifelines import CoxPHFitter\n",
    "from lifelines.utils import concordance_index as cindex\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from util import load_data\n",
    "\n",
    "from public_tests import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vZMwq0VfW5TW"
   },
   "source": [
    "<a name='2'></a>\n",
    "## 2. Load the Dataset\n",
    "\n",
    "Run the next cell to load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = load_data('data/pbc.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3. Explore the Dataset\n",
    "\n",
    "In the lecture videos `time` was in months, however in this assignment, `time` will be converted into years. Also notice that we have assigned a numeric value to `sex`, where `female = 0` and `male = 1`.\n",
    "\n",
    "Next, familiarize yourself with the data and the shape of it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "T1a_aHGmXT_C",
    "outputId": "1bbcf6d9-f293-49f4-963a-827c8e79813b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(258, 19)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>status</th>\n",
       "      <th>trt</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>ascites</th>\n",
       "      <th>hepato</th>\n",
       "      <th>spiders</th>\n",
       "      <th>edema</th>\n",
       "      <th>bili</th>\n",
       "      <th>chol</th>\n",
       "      <th>albumin</th>\n",
       "      <th>copper</th>\n",
       "      <th>alk.phos</th>\n",
       "      <th>ast</th>\n",
       "      <th>trig</th>\n",
       "      <th>platelet</th>\n",
       "      <th>protime</th>\n",
       "      <th>stage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.095890</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>58.765229</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>14.5</td>\n",
       "      <td>261.0</td>\n",
       "      <td>2.60</td>\n",
       "      <td>156.0</td>\n",
       "      <td>1718.0</td>\n",
       "      <td>137.95</td>\n",
       "      <td>172.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>12.2</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.328767</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>56.446270</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>302.0</td>\n",
       "      <td>4.14</td>\n",
       "      <td>54.0</td>\n",
       "      <td>7394.8</td>\n",
       "      <td>113.52</td>\n",
       "      <td>88.0</td>\n",
       "      <td>221.0</td>\n",
       "      <td>10.6</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.772603</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>70.072553</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>176.0</td>\n",
       "      <td>3.48</td>\n",
       "      <td>210.0</td>\n",
       "      <td>516.0</td>\n",
       "      <td>96.10</td>\n",
       "      <td>55.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.273973</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>54.740589</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.8</td>\n",
       "      <td>244.0</td>\n",
       "      <td>2.54</td>\n",
       "      <td>64.0</td>\n",
       "      <td>6121.8</td>\n",
       "      <td>60.63</td>\n",
       "      <td>92.0</td>\n",
       "      <td>183.0</td>\n",
       "      <td>10.3</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5.019178</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>55.534565</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>4.09</td>\n",
       "      <td>52.0</td>\n",
       "      <td>824.0</td>\n",
       "      <td>60.45</td>\n",
       "      <td>213.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>9.7</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        time  status  trt        age  sex  ascites  hepato  spiders  edema  \\\n",
       "0   1.095890     1.0  0.0  58.765229  0.0      1.0     1.0      1.0    1.0   \n",
       "1  12.328767     0.0  0.0  56.446270  0.0      0.0     1.0      1.0    0.0   \n",
       "2   2.772603     1.0  0.0  70.072553  1.0      0.0     0.0      0.0    0.5   \n",
       "3   5.273973     1.0  0.0  54.740589  0.0      0.0     1.0      1.0    0.5   \n",
       "6   5.019178     0.0  1.0  55.534565  0.0      0.0     1.0      0.0    0.0   \n",
       "\n",
       "   bili   chol  albumin  copper  alk.phos     ast   trig  platelet  protime  \\\n",
       "0  14.5  261.0     2.60   156.0    1718.0  137.95  172.0     190.0     12.2   \n",
       "1   1.1  302.0     4.14    54.0    7394.8  113.52   88.0     221.0     10.6   \n",
       "2   1.4  176.0     3.48   210.0     516.0   96.10   55.0     151.0     12.0   \n",
       "3   1.8  244.0     2.54    64.0    6121.8   60.63   92.0     183.0     10.3   \n",
       "6   1.0  322.0     4.09    52.0     824.0   60.45  213.0     204.0      9.7   \n",
       "\n",
       "   stage  \n",
       "0    4.0  \n",
       "1    3.0  \n",
       "2    4.0  \n",
       "3    4.0  \n",
       "6    3.0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df.shape)\n",
    "\n",
    "# df.head() only outputs the top few rows\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zy5BmjCV-Uo2"
   },
   "source": [
    "Take a minute to examine particular cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "id": "01I3ChzL-T-f",
    "outputId": "68e209dc-7a44-434b-d44c-4a1e817ee6ca"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "time          11.175342\n",
       "status         1.000000\n",
       "trt            0.000000\n",
       "age           44.520192\n",
       "sex            1.000000\n",
       "ascites        0.000000\n",
       "hepato         1.000000\n",
       "spiders        0.000000\n",
       "edema          0.000000\n",
       "bili           2.100000\n",
       "chol         456.000000\n",
       "albumin        4.000000\n",
       "copper       124.000000\n",
       "alk.phos    5719.000000\n",
       "ast          221.880000\n",
       "trig         230.000000\n",
       "platelet      70.000000\n",
       "protime        9.900000\n",
       "stage          2.000000\n",
       "Name: 23, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = 20\n",
    "df.iloc[i, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pYZKl_9Tk2vS"
   },
   "source": [
    "Now, split your dataset into train, validation and test set using 60/20/20 split. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V4HJSZaMk1xG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of patients: 258\n",
      "Total number of patients in training set: 154\n",
      "Total number of patients in validation set: 52\n",
      "Total number of patients in test set: 52\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "df_dev, df_test = train_test_split(df, test_size = 0.2)\n",
    "df_train, df_val = train_test_split(df_dev, test_size = 0.25)\n",
    "\n",
    "print(\"Total number of patients:\", df.shape[0])\n",
    "print(\"Total number of patients in training set:\", df_train.shape[0])\n",
    "print(\"Total number of patients in validation set:\", df_val.shape[0])\n",
    "print(\"Total number of patients in test set:\", df_test.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before proceeding to modeling, let's normalize the continuous covariates to make sure they're on the same scale. Again, we should normalize the test data using statistics from the train data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "continuous_columns = ['age', 'bili', 'chol', 'albumin', 'copper', 'alk.phos', 'ast', 'trig', 'platelet', 'protime']\n",
    "mean = df_train.loc[:, continuous_columns].mean()\n",
    "std = df_train.loc[:, continuous_columns].std()\n",
    "df_train.loc[:, continuous_columns] = (df_train.loc[:, continuous_columns] - mean) / std\n",
    "df_val.loc[:, continuous_columns] = (df_val.loc[:, continuous_columns] - mean) / std\n",
    "df_test.loc[:, continuous_columns] = (df_test.loc[:, continuous_columns] - mean) / std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the summary statistics on our training dataset to make sure it's standardized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>bili</th>\n",
       "      <th>chol</th>\n",
       "      <th>albumin</th>\n",
       "      <th>copper</th>\n",
       "      <th>alk.phos</th>\n",
       "      <th>ast</th>\n",
       "      <th>trig</th>\n",
       "      <th>platelet</th>\n",
       "      <th>protime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "      <td>1.540000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>9.833404e-16</td>\n",
       "      <td>-3.258577e-16</td>\n",
       "      <td>1.153478e-16</td>\n",
       "      <td>1.153478e-16</td>\n",
       "      <td>5.767392e-18</td>\n",
       "      <td>1.326500e-16</td>\n",
       "      <td>-1.263059e-15</td>\n",
       "      <td>8.074349e-17</td>\n",
       "      <td>2.018587e-17</td>\n",
       "      <td>1.291896e-14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-2.304107e+00</td>\n",
       "      <td>-5.735172e-01</td>\n",
       "      <td>-1.115330e+00</td>\n",
       "      <td>-3.738104e+00</td>\n",
       "      <td>-9.856552e-01</td>\n",
       "      <td>-7.882167e-01</td>\n",
       "      <td>-1.489281e+00</td>\n",
       "      <td>-1.226674e+00</td>\n",
       "      <td>-2.058899e+00</td>\n",
       "      <td>-1.735556e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-6.535035e-01</td>\n",
       "      <td>-4.895812e-01</td>\n",
       "      <td>-5.186963e-01</td>\n",
       "      <td>-5.697976e-01</td>\n",
       "      <td>-6.470611e-01</td>\n",
       "      <td>-5.186471e-01</td>\n",
       "      <td>-8.353982e-01</td>\n",
       "      <td>-6.884514e-01</td>\n",
       "      <td>-6.399831e-01</td>\n",
       "      <td>-7.382590e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-6.443852e-03</td>\n",
       "      <td>-3.846612e-01</td>\n",
       "      <td>-2.576693e-01</td>\n",
       "      <td>5.663556e-02</td>\n",
       "      <td>-3.140636e-01</td>\n",
       "      <td>-3.416086e-01</td>\n",
       "      <td>-2.260984e-01</td>\n",
       "      <td>-2.495932e-01</td>\n",
       "      <td>-4.100373e-02</td>\n",
       "      <td>-1.398807e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.724289e-01</td>\n",
       "      <td>2.977275e-02</td>\n",
       "      <td>1.798617e-01</td>\n",
       "      <td>6.890921e-01</td>\n",
       "      <td>3.435366e-01</td>\n",
       "      <td>-4.620597e-03</td>\n",
       "      <td>6.061159e-01</td>\n",
       "      <td>3.755727e-01</td>\n",
       "      <td>6.617988e-01</td>\n",
       "      <td>3.587680e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2.654276e+00</td>\n",
       "      <td>5.239050e+00</td>\n",
       "      <td>6.243146e+00</td>\n",
       "      <td>2.140730e+00</td>\n",
       "      <td>5.495204e+00</td>\n",
       "      <td>4.869263e+00</td>\n",
       "      <td>3.058176e+00</td>\n",
       "      <td>5.165751e+00</td>\n",
       "      <td>3.190823e+00</td>\n",
       "      <td>4.447687e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age          bili          chol       albumin        copper  \\\n",
       "count  1.540000e+02  1.540000e+02  1.540000e+02  1.540000e+02  1.540000e+02   \n",
       "mean   9.833404e-16 -3.258577e-16  1.153478e-16  1.153478e-16  5.767392e-18   \n",
       "std    1.000000e+00  1.000000e+00  1.000000e+00  1.000000e+00  1.000000e+00   \n",
       "min   -2.304107e+00 -5.735172e-01 -1.115330e+00 -3.738104e+00 -9.856552e-01   \n",
       "25%   -6.535035e-01 -4.895812e-01 -5.186963e-01 -5.697976e-01 -6.470611e-01   \n",
       "50%   -6.443852e-03 -3.846612e-01 -2.576693e-01  5.663556e-02 -3.140636e-01   \n",
       "75%    5.724289e-01  2.977275e-02  1.798617e-01  6.890921e-01  3.435366e-01   \n",
       "max    2.654276e+00  5.239050e+00  6.243146e+00  2.140730e+00  5.495204e+00   \n",
       "\n",
       "           alk.phos           ast          trig      platelet       protime  \n",
       "count  1.540000e+02  1.540000e+02  1.540000e+02  1.540000e+02  1.540000e+02  \n",
       "mean   1.326500e-16 -1.263059e-15  8.074349e-17  2.018587e-17  1.291896e-14  \n",
       "std    1.000000e+00  1.000000e+00  1.000000e+00  1.000000e+00  1.000000e+00  \n",
       "min   -7.882167e-01 -1.489281e+00 -1.226674e+00 -2.058899e+00 -1.735556e+00  \n",
       "25%   -5.186471e-01 -8.353982e-01 -6.884514e-01 -6.399831e-01 -7.382590e-01  \n",
       "50%   -3.416086e-01 -2.260984e-01 -2.495932e-01 -4.100373e-02 -1.398807e-01  \n",
       "75%   -4.620597e-03  6.061159e-01  3.755727e-01  6.617988e-01  3.587680e-01  \n",
       "max    4.869263e+00  3.058176e+00  5.165751e+00  3.190823e+00  4.447687e+00  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.loc[:, continuous_columns].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BX3woHz-jit1"
   },
   "source": [
    "<a name='4'></a>\n",
    "## 4. Cox Proportional Hazards\n",
    "\n",
    "Our goal is to build a risk score using the survival data that we have. We'll begin by fitting a Cox Proportional Hazards model to your data.\n",
    "\n",
    "Recall that the Cox Proportional Hazards model describes the hazard for an individual $i$ at time $t$ as \n",
    "\n",
    "$$\n",
    "\\lambda(t, x) = \\lambda_0(t)e^{\\theta^T X_i}\n",
    "$$\n",
    "\n",
    "The $\\lambda_0$ term is a baseline hazard and incorporates the risk over time, and the other term incorporates the risk due to the individual's covariates. After fitting the model, we can rank individuals using the person-dependent risk term $e^{\\theta^T X_i}$. \n",
    "\n",
    "Categorical variables cannot be used in a regression model as they are. In order to use them, conversion to a series of variables is required.\n",
    "\n",
    "Since our data has a mix of categorical (`stage`) and continuous (`wblc`) variables, before we proceed further we need to do some data engineering. To tackle the issue at hand we'll be using the `Dummy Coding` technique. In order to use Cox Proportional Hazards, we will have to turn the categorical data into one hot features so that we can fit our Cox model. Luckily, Pandas has a built-in function called `get_dummies` that will make it easier for us to implement our function. It turns categorical features into multiple binary features.\n",
    "\n",
    "<img src=\"images/1-hot-encode.png\" style=\"padding-top: 5px;width: 60%;left: 0px;margin-left: 150px;margin-right: 0px;\">\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-1'></a>\n",
    "### Exercise 1 - to_one_hot\n",
    "In the cell below, implement the `to_one_hot(...)` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Remember to drop the first dummy for each each category to avoid convergence issues when fitting the proportional hazards model.</li>\n",
    "    <li> Check out the <a href=\"https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html\" > get_dummies() </a>  documentation. </li>\n",
    "    <li>Use <code>dtype=np.float64</code>.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VMzvx0xF_C3I"
   },
   "outputs": [],
   "source": [
    "# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def to_one_hot(dataframe, columns):\n",
    "    '''\n",
    "    Convert columns in dataframe to one-hot encoding.\n",
    "    Args:\n",
    "        dataframe (dataframe): pandas dataframe containing covariates\n",
    "        columns (list of strings): list categorical column names to one hot encode\n",
    "    Returns:\n",
    "        one_hot_df (dataframe): dataframe with categorical columns encoded\n",
    "                            as binary variables\n",
    "    '''\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    one_hot_df = pd.get_dummies(dataframe, columns = columns, drop_first = True, dtype=np.float64)\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return one_hot_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SGZfLeup_fUL"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['time', 'status', 'trt', 'age', 'sex', 'ascites', 'hepato', 'spiders', 'bili', 'chol', 'albumin', 'copper', 'alk.phos', 'ast', 'trig', 'platelet', 'protime', 'edema_0.5', 'edema_1.0', 'stage_2.0', 'stage_3.0', 'stage_4.0']\n",
      "There are 22 columns\n"
     ]
    }
   ],
   "source": [
    "# test cell ex1 - do not modify this test cell\n",
    "# List of categorical columns\n",
    "to_encode = ['edema', 'stage']\n",
    "\n",
    "one_hot_train = to_one_hot(df_train, to_encode)\n",
    "one_hot_val = to_one_hot(df_val, to_encode)\n",
    "one_hot_test = to_one_hot(df_test, to_encode)\n",
    "\n",
    "print(one_hot_val.columns.tolist())\n",
    "print(f\"There are {len(one_hot_val.columns)} columns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected Output:\n",
    "```\n",
    "One hot val columns:\n",
    "\n",
    " ['time', 'status', 'trt', 'age', 'sex', 'ascites', 'hepato', 'edema', 'bili', 'chol', 'albumin', 'copper', 'alk.phos', 'ast', 'trig', 'platelet', 'protime', 'spiders_1.0', 'stage_2.0', 'stage_3.0', 'stage_4.0'] \n",
    "\n",
    "There are 21 columns\n",
    "\n",
    " All tests passed.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you'll use the function you coded to transform the training, validation, and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_encode = ['edema', 'stage']\n",
    "\n",
    "one_hot_train = to_one_hot(df_train, to_encode)\n",
    "one_hot_val = to_one_hot(df_val, to_encode)\n",
    "one_hot_test = to_one_hot(df_test, to_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look for new features\n",
    "Now, let's take a peek at one of the transformed data sets. Do you notice any new features?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "w8EG8A9gXcpu",
    "outputId": "384d9ade-2c96-4979-d3b7-da2b8e50f2e0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(154, 22)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>status</th>\n",
       "      <th>trt</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>ascites</th>\n",
       "      <th>hepato</th>\n",
       "      <th>spiders</th>\n",
       "      <th>bili</th>\n",
       "      <th>chol</th>\n",
       "      <th>...</th>\n",
       "      <th>alk.phos</th>\n",
       "      <th>ast</th>\n",
       "      <th>trig</th>\n",
       "      <th>platelet</th>\n",
       "      <th>protime</th>\n",
       "      <th>edema_0.5</th>\n",
       "      <th>edema_1.0</th>\n",
       "      <th>stage_2.0</th>\n",
       "      <th>stage_3.0</th>\n",
       "      <th>stage_4.0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>279</th>\n",
       "      <td>3.868493</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.414654</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.300725</td>\n",
       "      <td>-0.096081</td>\n",
       "      <td>...</td>\n",
       "      <td>0.167937</td>\n",
       "      <td>0.401418</td>\n",
       "      <td>0.330031</td>\n",
       "      <td>0.219885</td>\n",
       "      <td>-1.137178</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>3.553425</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.069681</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.895363</td>\n",
       "      <td>0.406085</td>\n",
       "      <td>...</td>\n",
       "      <td>0.101665</td>\n",
       "      <td>0.472367</td>\n",
       "      <td>1.621764</td>\n",
       "      <td>-0.120868</td>\n",
       "      <td>-0.239610</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>4.846575</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.924494</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.510565</td>\n",
       "      <td>-0.225352</td>\n",
       "      <td>...</td>\n",
       "      <td>0.245463</td>\n",
       "      <td>1.899020</td>\n",
       "      <td>-0.580807</td>\n",
       "      <td>0.422207</td>\n",
       "      <td>0.159309</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>0.490411</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.938314</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.748475</td>\n",
       "      <td>-0.608191</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.650254</td>\n",
       "      <td>-0.288898</td>\n",
       "      <td>-0.481443</td>\n",
       "      <td>-0.727833</td>\n",
       "      <td>1.356065</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.328767</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.563645</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.405645</td>\n",
       "      <td>-0.210436</td>\n",
       "      <td>...</td>\n",
       "      <td>2.173526</td>\n",
       "      <td>-0.144699</td>\n",
       "      <td>-0.531125</td>\n",
       "      <td>-0.450972</td>\n",
       "      <td>-0.139881</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          time  status  trt       age  sex  ascites  hepato  spiders  \\\n",
       "279   3.868493     0.0  0.0 -0.414654  0.0      0.0     0.0      0.0   \n",
       "137   3.553425     1.0  0.0  0.069681  1.0      0.0     1.0      0.0   \n",
       "249   4.846575     0.0  1.0 -0.924494  0.0      0.0     1.0      0.0   \n",
       "266   0.490411     1.0  0.0  1.938314  0.0      1.0     1.0      1.0   \n",
       "1    12.328767     0.0  0.0  0.563645  0.0      0.0     1.0      1.0   \n",
       "\n",
       "         bili      chol  ...  alk.phos       ast      trig  platelet  \\\n",
       "279 -0.300725 -0.096081  ...  0.167937  0.401418  0.330031  0.219885   \n",
       "137  0.895363  0.406085  ...  0.101665  0.472367  1.621764 -0.120868   \n",
       "249 -0.510565 -0.225352  ...  0.245463  1.899020 -0.580807  0.422207   \n",
       "266  0.748475 -0.608191  ... -0.650254 -0.288898 -0.481443 -0.727833   \n",
       "1   -0.405645 -0.210436  ...  2.173526 -0.144699 -0.531125 -0.450972   \n",
       "\n",
       "      protime  edema_0.5  edema_1.0  stage_2.0  stage_3.0  stage_4.0  \n",
       "279 -1.137178        0.0        0.0        0.0        1.0        0.0  \n",
       "137 -0.239610        0.0        0.0        0.0        1.0        0.0  \n",
       "249  0.159309        0.0        0.0        0.0        0.0        1.0  \n",
       "266  1.356065        0.0        1.0        0.0        0.0        1.0  \n",
       "1   -0.139881        0.0        0.0        0.0        1.0        0.0  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(one_hot_train.shape)\n",
    "one_hot_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hNxuymLwyjqM"
   },
   "source": [
    "<a name='5'></a>\n",
    "## 5. Fitting and Interpreting a Cox Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ygiFcUKcAFQk"
   },
   "source": [
    "Run the following cell to fit your Cox Proportional Hazards model using the `lifelines` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "dDCS7p3xjbXB",
    "outputId": "41b12f82-8b35-43e1-d2a9-05258ac50b20"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<lifelines.CoxPHFitter: fitted with 154 total observations, 90 right-censored observations>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cph = CoxPHFitter()\n",
    "cph.fit(one_hot_train, duration_col = 'time', event_col = 'status', step_size=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5MUITR0QANDH"
   },
   "source": [
    "You can use `cph.print_summary()` to view the coefficients associated with each covariate as well as confidence intervals. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 903
    },
    "colab_type": "code",
    "id": "fH5AZs8vjcEv",
    "outputId": "5429f7d5-5669-431f-a014-cf609c90997f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>model</th>\n",
       "      <td>lifelines.CoxPHFitter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>duration col</th>\n",
       "      <td>'time'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>event col</th>\n",
       "      <td>'status'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>number of observations</th>\n",
       "      <td>154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>number of events observed</th>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>partial log-likelihood</th>\n",
       "      <td>-230.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time fit was run</th>\n",
       "      <td>2022-10-19 11:58:36 UTC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coef</th>\n",
       "      <th>exp(coef)</th>\n",
       "      <th>se(coef)</th>\n",
       "      <th>coef lower 95%</th>\n",
       "      <th>coef upper 95%</th>\n",
       "      <th>exp(coef) lower 95%</th>\n",
       "      <th>exp(coef) upper 95%</th>\n",
       "      <th>z</th>\n",
       "      <th>p</th>\n",
       "      <th>-log2(p)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>trt</th>\n",
       "      <td>-0.22</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.30</td>\n",
       "      <td>-0.82</td>\n",
       "      <td>0.37</td>\n",
       "      <td>0.44</td>\n",
       "      <td>1.45</td>\n",
       "      <td>-0.73</td>\n",
       "      <td>0.46</td>\n",
       "      <td>1.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.23</td>\n",
       "      <td>1.26</td>\n",
       "      <td>0.19</td>\n",
       "      <td>-0.13</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.88</td>\n",
       "      <td>1.82</td>\n",
       "      <td>1.26</td>\n",
       "      <td>0.21</td>\n",
       "      <td>2.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <td>0.34</td>\n",
       "      <td>1.41</td>\n",
       "      <td>0.40</td>\n",
       "      <td>-0.45</td>\n",
       "      <td>1.14</td>\n",
       "      <td>0.64</td>\n",
       "      <td>3.11</td>\n",
       "      <td>0.84</td>\n",
       "      <td>0.40</td>\n",
       "      <td>1.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ascites</th>\n",
       "      <td>-0.10</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.56</td>\n",
       "      <td>-1.20</td>\n",
       "      <td>1.01</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.75</td>\n",
       "      <td>-0.17</td>\n",
       "      <td>0.86</td>\n",
       "      <td>0.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hepato</th>\n",
       "      <td>0.31</td>\n",
       "      <td>1.36</td>\n",
       "      <td>0.38</td>\n",
       "      <td>-0.44</td>\n",
       "      <td>1.06</td>\n",
       "      <td>0.64</td>\n",
       "      <td>2.89</td>\n",
       "      <td>0.81</td>\n",
       "      <td>0.42</td>\n",
       "      <td>1.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spiders</th>\n",
       "      <td>-0.18</td>\n",
       "      <td>0.83</td>\n",
       "      <td>0.38</td>\n",
       "      <td>-0.94</td>\n",
       "      <td>0.57</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.77</td>\n",
       "      <td>-0.47</td>\n",
       "      <td>0.64</td>\n",
       "      <td>0.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bili</th>\n",
       "      <td>0.05</td>\n",
       "      <td>1.05</td>\n",
       "      <td>0.18</td>\n",
       "      <td>-0.29</td>\n",
       "      <td>0.39</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.48</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.77</td>\n",
       "      <td>0.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>chol</th>\n",
       "      <td>0.19</td>\n",
       "      <td>1.20</td>\n",
       "      <td>0.15</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>0.47</td>\n",
       "      <td>0.91</td>\n",
       "      <td>1.60</td>\n",
       "      <td>1.28</td>\n",
       "      <td>0.20</td>\n",
       "      <td>2.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>albumin</th>\n",
       "      <td>-0.40</td>\n",
       "      <td>0.67</td>\n",
       "      <td>0.18</td>\n",
       "      <td>-0.75</td>\n",
       "      <td>-0.06</td>\n",
       "      <td>0.47</td>\n",
       "      <td>0.94</td>\n",
       "      <td>-2.28</td>\n",
       "      <td>0.02</td>\n",
       "      <td>5.46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>copper</th>\n",
       "      <td>0.30</td>\n",
       "      <td>1.35</td>\n",
       "      <td>0.16</td>\n",
       "      <td>-0.01</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.99</td>\n",
       "      <td>1.84</td>\n",
       "      <td>1.91</td>\n",
       "      <td>0.06</td>\n",
       "      <td>4.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>alk.phos</th>\n",
       "      <td>-0.22</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.14</td>\n",
       "      <td>-0.49</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.61</td>\n",
       "      <td>1.05</td>\n",
       "      <td>-1.62</td>\n",
       "      <td>0.11</td>\n",
       "      <td>3.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ast</th>\n",
       "      <td>0.21</td>\n",
       "      <td>1.24</td>\n",
       "      <td>0.16</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>0.53</td>\n",
       "      <td>0.91</td>\n",
       "      <td>1.69</td>\n",
       "      <td>1.34</td>\n",
       "      <td>0.18</td>\n",
       "      <td>2.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trig</th>\n",
       "      <td>0.20</td>\n",
       "      <td>1.23</td>\n",
       "      <td>0.16</td>\n",
       "      <td>-0.11</td>\n",
       "      <td>0.52</td>\n",
       "      <td>0.89</td>\n",
       "      <td>1.68</td>\n",
       "      <td>1.27</td>\n",
       "      <td>0.21</td>\n",
       "      <td>2.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>platelet</th>\n",
       "      <td>0.14</td>\n",
       "      <td>1.15</td>\n",
       "      <td>0.15</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.86</td>\n",
       "      <td>1.54</td>\n",
       "      <td>0.92</td>\n",
       "      <td>0.36</td>\n",
       "      <td>1.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protime</th>\n",
       "      <td>0.36</td>\n",
       "      <td>1.43</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.69</td>\n",
       "      <td>1.03</td>\n",
       "      <td>1.99</td>\n",
       "      <td>2.15</td>\n",
       "      <td>0.03</td>\n",
       "      <td>4.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>edema_0.5</th>\n",
       "      <td>1.24</td>\n",
       "      <td>3.47</td>\n",
       "      <td>0.46</td>\n",
       "      <td>0.35</td>\n",
       "      <td>2.14</td>\n",
       "      <td>1.42</td>\n",
       "      <td>8.50</td>\n",
       "      <td>2.72</td>\n",
       "      <td>0.01</td>\n",
       "      <td>7.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>edema_1.0</th>\n",
       "      <td>2.02</td>\n",
       "      <td>7.51</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.84</td>\n",
       "      <td>3.20</td>\n",
       "      <td>2.31</td>\n",
       "      <td>24.43</td>\n",
       "      <td>3.35</td>\n",
       "      <td>&lt;0.005</td>\n",
       "      <td>10.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>stage_2.0</th>\n",
       "      <td>1.21</td>\n",
       "      <td>3.35</td>\n",
       "      <td>1.08</td>\n",
       "      <td>-0.92</td>\n",
       "      <td>3.33</td>\n",
       "      <td>0.40</td>\n",
       "      <td>28.06</td>\n",
       "      <td>1.11</td>\n",
       "      <td>0.27</td>\n",
       "      <td>1.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>stage_3.0</th>\n",
       "      <td>1.18</td>\n",
       "      <td>3.27</td>\n",
       "      <td>1.09</td>\n",
       "      <td>-0.96</td>\n",
       "      <td>3.33</td>\n",
       "      <td>0.38</td>\n",
       "      <td>27.86</td>\n",
       "      <td>1.08</td>\n",
       "      <td>0.28</td>\n",
       "      <td>1.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>stage_4.0</th>\n",
       "      <td>1.41</td>\n",
       "      <td>4.10</td>\n",
       "      <td>1.15</td>\n",
       "      <td>-0.85</td>\n",
       "      <td>3.67</td>\n",
       "      <td>0.43</td>\n",
       "      <td>39.43</td>\n",
       "      <td>1.22</td>\n",
       "      <td>0.22</td>\n",
       "      <td>2.18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Concordance</th>\n",
       "      <td>0.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Log-likelihood ratio test</th>\n",
       "      <td>97.63 on 20 df, -log2(p)=38.13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cph.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:**\n",
    "\n",
    "- According to the model, was treatment `trt` beneficial? \n",
    "- What was its associated hazard ratio? \n",
    "    - Note that the hazard ratio is how much an incremental increase in the feature variable changes the hazard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Check your answer!</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "<ul>\n",
    "    <li>You should see that the treatment (trt) was beneficial because it has a negative impact on the hazard (the coefficient is negative, and exp(coef) is less than 1).</li>\n",
    "    <li>The associated hazard ratio is ~0.8, because this is the exp(coef) of treatment.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare the predicted survival curves for treatment variables. Run the next cell to plot survival curves using the `plot_covariate_groups()` function. \n",
    "- The y-axis is th survival rate\n",
    "- The x-axis is time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "colab_type": "code",
    "id": "Uxl0icyBS4Dr",
    "outputId": "5fa08369-e89e-424f-f9f0-60cf7a1cfbcd"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3df1yUZb7/8deV2qrlr6TSFfyVRpp6sDDWbHMq3DQNKy2IbBPb6FuaWdlJ9+SPrE062jnqZj+0Fd08rrZqpVZqbqKZZlBLK6KWaSmupmhqairG9f1jYBxggAFmGGZ4Px8PHnHPfXPPZ9I+XVz35/pcxlqLiIgEvwsCHYCIiPiGErqISIhQQhcRCRFK6CIiIUIJXUQkRNQN1BuHhYXZtm3bBurtRUSC0hdffJFrrb3U07mAJfS2bduSkZERqLcXEQlKxpjvSzunKRcRkRChhC4iEiKU0EVEQkTA5tBFpGry8vLIycnh9OnTgQ5F/KB+/fqEh4dTr149r39GCV0kSOXk5NCoUSPatm2LMSbQ4YgPWWs5fPgwOTk5tGvXzuufK3fKxRgzxxhz0BiTVcp5Y4yZYYzZaYz5lzHmmgrELSKVdPr0aZo3b65kHoKMMTRv3rzCv315M4c+F+hbxvl+QMeCr2TgtQpFICKVpmQeuirzZ1tuQrfWrgeOlHHJQOCv1ukzoKkxpmV599225Z+Mffhu7yMVEZEy+aLKpRWw1+04p+C1EowxycaYDGNMxi+/5BP5wwoyn7+eHu2bkHLPlWz++8s+CEdEqsvRo0d59dVXK3yuNLt37yYmJoYOHToQHx/P2bNnfRFmrVGtZYvW2lnW2mhrbXS7NuH0uO5a17nwc3vpmvkcPdo30chdJEiUlrTPnTtXqYT+zDPP8MQTT7Bz506aNWvGX/7yF1+FWiv4osplHxDhdhxe8FqZLmp2OVf/cQMA6eNg899fJnvrYs79/DlRh9/n2MzfMfDPXzAyti13XduC3J/OsvTAr0l+cZ4PQhYRXxgzZgzffvstUVFR1KtXj/r169OsWTO2b9/ONddc4zrXp08fpkyZUua9rLV8/PHHLFiwAIAHHniAiRMn8sgjj1THRwkJvkjoy4ARxpiFQAxwzFq7v6I3ibn7Kbj7KR5pMo72x9aw7YfjnDx7jj0/nmLr/mOkrkhnU84vJHfMha6DITrJB6GLhIbnlm8l+9/HfXrPzr9uzITbry7zmpSUFLKyssjMzCQtLY3+/fuTlZVFu3bt+O6771znAH766Sd++9vferzPggULuOyyy2jatCl16zrTUnh4OPv2lTs2FDflJnRjzN8ABxBmjMkBJgD1AKy1rwMfALcBO4FTQJUy7cXXP8TUzNsAaP8MbML5dbzpOO5ovZcDW9fD1vW02LJYiV2khrnuuutKrZtu1KiRK7l7kpub66+wao1yE7q19t5yzltguK8CSoxpTWJM6xKvL4iaw3uZ+/jzi/cQZo7x5cUb2Puv9dw/ZDTPDuhA7NVhJW+mhC+1RHkj6epy0UUXlXquvBF6p06dOHr0KOfOnaNu3brk5OTQqpXH+gopRdCsFC1M9H/4bjhfn/wVY/P28pv8tZw8u4vvjpzk8911+M+3d/DcHR3pHdmcHV+sY93Sj0m+a/H5myjBi/hUo0aN+Omnn7w6V94IHeCmm25i8eLFJCQkMG/ePAYOHOjTeENd0CT0Qm9OHgPAgs17WJB5N+07wwu7j5B38AB5+TOZmPd7Ljvbnb2f59Do53+TDGTl/MSIv2wksatbgldyF6my5s2b06tXL7p06UKDBg24/PLLPZ7r169fuQ9FAV566SUSEhJ49tln6d69Ow8++KA/ww85xjljUv2io6Otrza4WLB5D+9lFn14suadBdQLa0Pv3/bi2L5vOfjun3ixz8W0aFyfUambSOxaj+S7bj7/A0rwEmS2bdtGp06dAh2G+JGnP2NjzBfW2mhP1wfdCN0TT/PuC6JauZJ8k1ZXsP22qfw/4Mq8XE42O8IVnS4iffdRnn57O6/cfIYu3ztLKJXURSRYhURC96R4kj8/ir+Er2/9bx4EOuYdIDt/Jn+/uAfH984n67lHNecuIkErZBN6ce4J3j253/TUTObtPsKCT9ZwbbNTJAPrdhxmwsKMonPuxSnZi0gNU2sSujtPo/fHMm/jk4guxJ/txsG8f/JLy7NEda/Pmq25vLBiJ2899B9EXNKAlVsOkbL0Sz1gFZEap1Ym9OISY1rDc0+75twvu7I72Y0m83LLxjx0xUnIeAHumQcREbByJVH7XyYuxrI88wf279pK8vcbYIumakQksJTQCxQftce/sYns/ceZTWMuv3cyoz/IwdlIsgk4JvFxVCtm/v1eerbrDW3qnL/R9xucX6CkLiLVSptEl2JgVCs6t2zs8dzm3Uf44ztbONPuBjren8LSJg/imHeSY3ctgAHTnBetGAWp/Yt+ZaRW4ycQ8T9ft8995ZVX6NChA8YYtQKoBI3QS1FaCwI4/1B1Mzcyee0+7mx6yHVu0bcNSflbUx75TWOS27j9kEbuEoIKk/ajjz5a5HX39rnFz5WlV69eDBgwAIfD4eNIa4eQWFgUKAs27+GP72wBIKbdJQB8+8m7tP7Vaeb/eTJLly7lzTff5KOPPqLeV/Odo3aANjecv4nm26WSasLCooSEBN577z0iIyM9ts8tPOdN+1x3bdu2JSMjg7AwDz2aapFaubAoUApH8O6rVHPDbyQXeHdL0V8X52YZUuY2ZPsEtz20C0ftWzyURirRS0V8OAYObPHtPVt0hX4pZV7iy/a5nTt39m38tZASehV5KoH84ztb+OM7W4hpdwWX3zuZIXMy+PaTbVxwSVtIep/Zs2ezaNEi1qRM85zMleglSFWlfa5UnRK6j3katYNz5E74jSzYvOf8i9FJvPr5zyxfvpwPP/wQgOnTp/OPj//NsqRfl7y55uGlNOWMpKtLVdrnaoRedUrofuCxt0yRkXsXmt/dhfg3NtFw95GSN2gaAUnLmDp1Kps2bWLJkiUApIwYzCXfv09ym8VK6FIj+Lp9rlSNyharSWJMa168s6vr4SlA9v7jnOpwi2t0DvD444+zbNkyj/eYuyaLjw82c86VqhRSagD3FrlPP/201+dKM2PGDMLDw8nJyaFbt2784Q9/8EfYIUtVLgFUuHjJU737wKhWnssmM1LPz60f2OJ8cJX0vp8jlZqoJlS5iH+pyiWIDIzyvL3W5t1H2Lz7SIl5eID8L7PoFv4bxo0b5xyhF47WPdEDVJFaRQk9gErdP9XDhh3gTPS5G75kf0RTxgHjP61DeF4Tklt4uHlZlTLFKfGLhAQl9BqorET/R0bTqmAe/u0NX9OzZ0+Sk1IZO3Yshw8fZtasWQCMHtKXK+vsK7pa1ZPC2mUldJGgp4QeRBJjWhcZuW/fvr3Ua1dkfMctt9wCSTPLvqmnaRuN2EWCkhJ6EMref5z4NzYB5x+eTp48ucg1hcl+1Chnu4Fp05xNw4YPH06DBg2YOnUqAMnLThDd2G3aRiN2kaClhB5k3B+kZu8/DlBqEzFvrN/2A83vvBOSCv6HoBG7SNBS2WIQK6vsEcooffQgKSmJiIgIJsW1K/ogVaWRNVagyxa/++47BgwYQFZWll/un5aWxtSpU1mxYgXLli0jOzubMWPG+OW9qiojI4O//vWvzJgxo8I/O3fuXDIyMnjllVdKnFPZYi1SWtkjlF36WOZ9opOKjsaLj9g1WpcAiIuLIy4uLqAxnDt3jrp1PafM6OhooqM95thqpZWiQSwxpjWLHu7p8av4qtTSZO8/znuZ+0hNTWXSpEkMGTKEq666ynU+4W+5zNrexHlwYIt3ZZBSa5w7d4777ruPTp06MXjwYE6dOgXApEmT6NGjB126dCE5OZnCmYAZM2bQuXNnunXrRkJCAgAnT55k2LBhXHfddXTv3p333nuvxPvMnTuXESNGADB06FBGjhzJ9ddfT/v27Vm8+PzfySlTptCjRw+6devGhAkTStznl19+YejQoXTp0oWuXbvyv//7vwA4HA4KZwxyc3Np27at633j4uK4+eabueWWW0hISOD998//tjp06FAWL15MWloaAwYMID8/n7Zt23L06FHXNR07duSHH35g+fLlxMTE0L17d2JjY/nhhx8q/e+9VNbagHxde+21VgLvntc32i4TVtp7Xt9o73l9o732vv+0g5OfdJ2PjIy0kydPdh7Muc3aFyOc/yz+lT4nQJ+g9srOzi5y3Lt3b5uammqttfbs2bO2d+/e9q233rLWWnvy5Enbu3dvu3DhQmuttUePHrW9e/e2S5YssdZae+jQIdu7d2+7bNkya621+/fvL/f9d+/ebQG7YcMGa621SUlJdsqUKdZaaw8fPuy6bsiQIa77tmzZ0p4+fdpaa+2PP/5orbV27Nixrjh//PFH27FjR3vixAm7du1a279/f2uttampqXb48OHWWmsfeOABO3jwYPvLL7/YrVu32iuuuMJaa+2qVavsQw89ZPPz8+0vv/xi+/fvb9etW1ck5oyMDBsbG+s6Loyhd+/eNj093fXvok2bNq73bdWqlevzLF261P7+97+31lp75swZGx4ebk+dOlUk1pEjR9o5c5z/PXz22Wf2lltusdZae+TIEZufn2+ttXb27Nn2ySefLPHZiiv+Z2yttUCGLSWvaoReyxXfau9Mh5u54JrBruPt27czZswYBg0axNTMi53z6cVp5F5rRURE0KtXLwCGDBnChg3ObqBr164lJiaGrl278vHHH7N161YAunXrxn333cf8+fNd0xerV68mJSWFqKgoHA4Hp0+fZs+ePZ7fsMAdd9zBBRdcQOfOnV0j3dWrV7N69Wq6d+/ONddcw/bt2/nmm2+K/Fz79u3ZtWsXjz32GCtXrqRxY8/Pn9z16dOHSy5x/rbbr18/1q5dy5kzZ/jwww+58cYbadCgQZHr4+PjWbRoEQALFy4kPj4egJycHG699Va6du3KlClTXP9OfElz6LWcp82xSxXRA5JGl3xd8+w1Qlpamuv7evXqFTlu2LBhkeMmTZoUOQ4LCyty3KKFp+XHJRljShyfPn2aRx99lIyMDCIiIpg4cSKnT58G4P3332f9+vUsX76cP/3pT2zZsgVrLUuWLCEyMrLIvcqakvjVr37l+t4WTOdYaxk7diwPP/xwqT/XrFkzvvrqK1atWsXrr7/O22+/zZw5c6hbty75+fkArlgLubcErl+/Pg6Hg1WrVrFo0SLXtJG7nj17snPnTg4dOsS7777Ls88+C8Bjjz3Gk08+SVxcHGlpaUycOLHUOCtLI3QpobDOvfBrweY9LFmyhNGjRxMXF8f06dNd1/br149Xt19yfuSu0XqtsmfPHjZtcg4CFixYwA033OBKiGFhYZw4ccI1x52fn8/evXu56aabeOmllzh27BgnTpzg1ltv5c9//rMrMf/zn/+sVCy33norc+bM4cSJEwDs27ePgwcPFrkmNzeX/Px8Bg0axAsvvMCXX34JOLe8++KLLwCKzMl7Eh8fT2pqKp988gl9+/Ytcd4Yw5133smTTz5Jp06daN68OQDHjh2jVStnAcK8efMq9RnLoxG6FFG8csarWvc2PSGpYCNgNQyrVSIjI5k5cybDhg2jc+fOPPLIIzRs2JCHHnqILl260KJFC3r06AE4H0gOGTKEY8eOYa1l5MiRNG3alHHjxjFq1Ci6detGfn4+7dq1Y8WKFRWO5Xe/+x3btm2jZ8+eAFx88cXMnz+fyy67zHXNvn37SEpKco3GCxfkjR49mnvuuYdZs2bRv38pf3fd3uf+++9n4MCBXHjhhR6viY+Pp0ePHsydO9f12sSJE7n77rtp1qwZN998M7t3767wZyyP6tClTMVr3cutbXdv71ucatp9KtB16OJ/qkMXn/JmZWpsbCw5OTnOdgPF69jdlTZqFxGfUEKXMrk/NC0crRd/cHq0xbXc0KMP4KznhaIP6IrwNB2jaRgRn9BDUfFa8RLHQmc63MxPbW8EnAsthg4dSl5eHg6Hg/nz5wNw6tQpHK98e36RUiE9RK2SQE2Ziv9V5s9WI3TxWml92t1H7EOHDgUgLy+vxHXH8hvQ5LYxUFCXC5T9EFUj9zLVr1+fw4cP07x58xLlgxLcrLUcPnyY+vXrV+jnlNDFJzy19C1eB+2xHK3r4JKvgdr4eqFwM+VDhw4FOhTxg/r16xMeHl6hn1FClyrztqXvsWPHSElJoUePHtx1113k5uYyePQ8EhMTSU5OLnqxHqCWq169erRr1y7QYUgN4lVCN8b0BaYDdYA3rbUpxc63BuYBTQuuGWOt/cDHsUoN5c2DU4BvP3mXU9nrXXXJ4Jyaadmypecbl1XP7k5TMyKAFwndGFMHmAn0AXKAdGPMMmtttttlzwJvW2tfM8Z0Bj4A2vohXqnhymrpe6bDzXT97R3cdZdz4UdYWBiffvqp54tLm4opTlMzIi7ejNCvA3Zaa3cBGGMWAgMB94RugcLyhybAv30ZpASP0h6cguc+MQcOHCAlJYW+ffvSt29f9u7dy/3338+zzz5LrDcLkDQ1I+LiTUJvBex1O84BYopdMxFYbYx5DLgIiPV0I2NMMpAM0Lp15bdNk+BVfDrGOQ2z2WNPDK8VTs1o6kVqOV/Vod8LzLXWhgO3AW8ZY0rc21o7y1obba2NvvTSS3301hIsPNWxn+lwM10f/h9XQo+IiCAtLc3V5rRcXQc72wmonl3Eq4S+D4hwOw4veM3dg8DbANbaTUB9IMwXAUro8LTDUmn7oQ4bNozhw4cDsGPHDhwOBxs3bgQgKysLh8NBenq6c0Se9L7nPu0itYw3Uy7pQEdjTDuciTwBSCx2zR7gFmCuMaYTzoSu4liptMTExBL9sT3JzMwEIMrfAYkEgXITurX2nDFmBLAKZ0niHGvtVmPMJJxbIS0DngJmG2OewPmAdKjVmmTxkucyx65cXN9ZMRMZGVlkkVKXLl1cx67eMQ9c5H2ZI2i+XUKSV3XoBTXlHxR7bbzb99lAL9+GJrVBaWWOXvVhB6ZNm+b85lwFNkVQqaOEKK0UlYDypj9MWaKiokhPT8fxdCkrTj1RqaOEKCV0qbFKW3FaXJcLcwkPD+f666/3/uYqdZQQpIQuNVJZK07dZe8/Di3DWFTQptcrhatQNfUiIUZb0ElQKxzBP97VkpKSwpQpU4iMjGTdunVMmDCBOXPm0L59e9asWcMLL7zAW2+9RUREQRVu4dSLtsSTIKIt6CTkZWVlcfz4ca+uXblyJQB9oWhljKZfJMhphC5BzX0T63I3sC7gKnWc+sD51aXawFqChEboErIK59q9LXMEWLhwofObFi3Oj8hV+SIhQHuKSlArbCdQWgsBT1q0aOEsdXQ4mDVrlh+jE6leGqFLrRUdHc1dd911/oWyVppqfl2CgBK6hAxv69YBBkb9B1On3n7+hbI21FB5owQJJXQJCd7WrcP5+fb6+zJISUnho48+okl0UukJW/PrEiSU0CUklLVTUnGFo/jc3FwaNmwIwKJFi5zn4uM9/9D3GyAjVaN0qdH0UFRqreTkZNLS0mjSpAmvvfYar732mucLC6djtIGG1HAaoUutVDjfXli7/sEHH5R+cXSSkrkEBY3QpdYp3Aove/9x3st0br7VsGFDli5disPhIC8vD4C5c+e6FiGJBAON0KXWKZxv97YixkUbaEgNp6X/Umu5tw0ASm0dMHv2bPjuUx7q4OWuimojIH6kpf8iHriXOpbVOqCwAuahP63x7sYqc5QAUUKXWsu91LGs6Zc1a7xM5CIBpoQu4oVXX30VgEcffdS7H1BbXgkAVbmIeGH58uUsX77cu4u7DnbOoYMzsavkUaqJRugiBcrqBXP/xDe8XomKexsBzadLNdIIXYTztemeFNarT58+nbi4ONfrU6dOZdCgQa7jlJQUEhIS/B6rSGk0Qheh7F4wFa5XL059YKSaqA5dpByFCX3Rwz29uv75558HYNy4cc5EvmIUtLlBdeniE2XVoWvKRcQLm3cfYcHmPV5du2PHDnbs2OE8iE5yJnORaqApF5FyDIxqxebdR3gvc59XD0bnz58PwPjx49m7dy+pN/o7QhEnJXSRciTGtHY18aq0ivSBqS6qjw85SugifjJp0iTnNxmpgQ3EE22rF5KU0EW8VLE9S52NvsaOHQvA5Mk17IFoTfttQXxCCV3EC5XZszQxpjWHDx/2V0giJSihi3ihMnuWAsyaNctfIYmUoLJFET8bPXo0o0ePDnQYJRUueJKQoYQu4geF8+3xb2ziq+8O8vPPPwc6pKK08XVI0pSLiI8V3zijc59HmOnlKtNqo42vQ5JG6CI+lhjTmkUP92TRwz1dDb9GjRrFqFGjXNcMHz68yDRMcnKyqyIGICkpifHjx7uOhwwZ4mopAJCQkEBKSorreNCgQUydOtV1HBcXx/Tp013H/fr1c/V0B4iNjWVWmncrXyV4KKGL1EI5OTkYE+goxNfUnEvEjyra2KtapfY/v6F1cVpFWmOpOZdIAFWksVe1ct9ZyZ12WQpaXj0UNcb0BaYDdYA3rbUpHq65B5gIWOAra22iD+MUCUoVbexVXRwOBwBpaWklT2oVadAqN6EbY+oAM4E+QA6QboxZZq3NdrumIzAW6GWt/dEYc5m/AhYJJj5p7OUHQ4cODXQI4gfejNCvA3Zaa3cBGGMWAgOBbLdrHgJmWmt/BLDWHvR1oCLiO0OHDiUvL4+8vDzq1asX6HDER7yZQ28F7HU7zil4zd2VwJXGmE+NMZ8VTNGUYIxJNsZkGGMyDh06VLmIRcQn+vTpQ58+fQIdhviQrxYW1QU6Ag4gHFhvjOlqrT3qfpG1dhYwC5xVLj56b5EaryKdGqtLZ8cdXN8hzPNJT/3bVflS43mT0PcBEW7H4QWvucsBNltr84DdxpivcSb4dJ9EKRLEKtKpsbpk7z9O55YxDBnioZyysC2AO/VPDwreJPR0oKMxph3ORJ4AFK9geRe4F0g1xoThnILZ5ctARYJVRTo1Vpf4NzZx7uxpxo8fz9VXX018fDzHjh1j4MCBjBw5kruKb2itypegUO4curX2HDACWAVsA9621m41xkwyxsQVXLYKOGyMyQbWAk9ba9UIWqQG+37zStavXx/oMMSHtFJUpBYqawVrbm4uAGFhbvPrhSP04iN3qXZaKSoiJZS2gnXw4MEMHlzKPHpqf+eX+qjXSGqfK1ILlbWC9amnnir5A+4PSvWAtMZSQhephcpawXr77beXfDE66XwC1wPSGktTLiJSxIEDBxg1ahQrV64EYO/evTgcDu2PGgSU0EWkiGXLlpGZmVnktbp169K+ffsARSTe0pSLiBSRnJxMcnKy6zgiIoI1a9YEMCLxlhK6SC1WWkuCgVGtijws3bXLuU7QNUr31BqgOLUKqHZK6CK1VGktCbL3HwcoktCHDRsGFPRP99QaoDhVwgSEErpILVVaSwJPI/bnnnvu/IF7xUtpVAkTEHooKiLl6t27Ny1atMDhcLBx48ZAhyOlUEIXEQkRSugi4pXIyEjS0tJo3LgxWVlZgQ5HPNAcuoiU4Kn6pbDyZcSIEUApG0xLQCmhi0gRnqpf3CtfpkyZUt0hiZeU0EWkCE/VL+6j9R49enh3o+K16qpL9zvNoYtIhWRmZpKQkOBqD5Ceno7D4XDNq2/cuBHHK98ya3uT8z90YAtsWRyIcGsVjdBFpEI+//xzDhw4UOY17br1pPeYMRAZ6XxBdenVQjsWiUi5ytrhyCva8chnytqxSCN0EfGKe+VL8V4vxa1btw5wLkiS6qOELiLlcq988dTrpbgJEyYAKm2sbkroIlIu98oXT71eipszZ46/QxIPlNBFxOc8bobhTctdUHljFSihi4jPFW6IERsb63zBm5a7oLa7VaSELiI+98ILL3D11VcTGxvLypUrSUmZR2JiYpGdkDxSeWOVKKGLiM8lJibSr18/13FUVBRxcXEBjKh2UEIXkQorbeu687py8b8tiRHQt29f+vbtW22x1WZK6CJSIaVtXeeueGnj8uXLAbj99tv9F5gooYtIxZS2dZ274qP3l19+GVBC9zcldBHxu8WL1ZirOiihi4jfhYWFBTqEWkHtc0XE75YuXcrSpUsDHUbIU0IXEb+bMWMG6enpACxatAiHw8GpU6cAmD9/Pg6Hg7y8vECGGBI05SIifpeYmEh8fHy5183dkAPAUC0UrRT1QxcRn4t/YxPZ+4/TuWVj12vltdwFcFzVHIC07Yf9Gl8wUz90EalWxWvVvWm5C5D2zG/8FlNtoIQuIj5XvFbdm5a7UnV6KCoiNcbsdXuYvW5PoMMIWhqhi0iNsejz/QA8FOA4gpUSuojUGGuejgl0CEHNq4RujOkLTAfqAG9aa1NKuW4QsBjoYa1VCYuIVJy3Oxt5qxbtgFRuQjfG1AFmAn2AHCDdGLPMWptd7LpGwOPAZn8EKiLBrfyWu1Dno3y6mQaMuc1Hb1rLdkDyZoR+HbDTWrsLwBizEBgIZBe77nngJeBpn0YoIkHPm5a7AMu/ymXtry5hzN/f980b17IdkLxJ6K2AvW7HOUCRiS5jzDVAhLX2fWNMqQndGJMMJAO0bl12PaqIhA5vWu4CxDOjGqIJXVUuWzTGXAD8D/BUeddaa2dZa6OttdGXXnppVd9aRETceDNC3wdEuB2HF7xWqBHQBUgzxgC0AJYZY+L0YFREKuLrfyxyfvNwz8AGEqS8SejpQEdjTDuciTwBSCw8aa09BriaHRtj0oDRSuYiUlEHd3wR6BCCWrkJ3Vp7zhgzAliFs2xxjrV2qzFmEpBhrV3m7yBFpHa44dH/DnQIQc2rOnRr7QfAB8VeG1/KtY6qhyUiIhWllaIiUmNsX73A+Y3m0CtFzblEpMY4vDuLw7uzAEhJSSEhIcF17vnnn2fIkCGu4/Hjx5OUdH7B0NixY0lOTnYdjx49muFvZVVD1DWHRugiUmP0evjFQIcQ1LRjkYjUGIWtARb5asoltT98vwEGTAuZ5f9l7VikKRcRCVmjVp9l1MrTsGVxoEOpFkroIhK6Lr8aGv060FFUG82hi8XBuoEAAAfPSURBVEjImjZtGqR+E+gwqo1G6CIiIUIJXURC1vDhw2tV6aKmXEQkZDVo0AAurBPoMKqNErqIhKypU6dC6rZAh1FtlNBFpEbxZqu6iphx5jSXN6rvs/vVZJpDF5EaY2BUKzq3bOyz+30063lGvLXVZ/er6TRCF5Eaw9ut6rzV6Z0wmvxce9Jc7fmkIlLrdLvzEZ44vCvQYVQbJXQRCWnPLvmGppwglf5Vu1HXwTW+H4wSuoiEtLzmVxJR/0DVbnJgi/OfSugiIoFz0aD/ZlJVuzemVnF0X01U5SIiEiKU0EUkpH02Z2KRnY5CmaZcRCSkNbq8DZGREYEOo1oooYtISLu6fxLjasmm05pyEREJEUroIhLSVkx7hp594gIdRrVQQheRkDUwqhUXXt6ekxe3AmDQoEHODowF4uLimD59uuu4X79+vPrqq67j2NhYZs+eXX0BV5Hm0EUkZCXGtOa9ex8OdBjVxlhrA/LG0dHRNiMjIyDvLSK1R2Er3kVVeDDquKo5AGnbD/skpqowxnxhrY32dE4jdBGRcgztFR7oELyihC4iUo6hNyihi4iEhLxz+fBDFvVqeE8XVbmIiJSjz+u76TP/VKDDKJdG6CIi5fjDE886v6kJPWGGmVJPKaGLiJQjWJp7KaGLSMjL3n/cVb5YGefOnmZAt1+T1DvSh1H5nhK6iIS0gVGtqnyPlS8/zicX1iXp6y99EJH/KKGLSEhLjGlNYkzrKt2jZ8bdPorGv5TQRUTK0To6NtAheEUJXUSkHGd/PhHoELyiOnQRkXJ8+uozrJ/2OAC5ubk4HA6WL18OwIEDB3A4HKxcuRKAvXv34nA4WLNmDQC7du3C4XCwbt06AHbs2IHD4WDjxo0AZGVl4XA4SE9PByAzMxOHw0FmZiYA6enpOBwOsrKyyo3Tq4RujOlrjNlhjNlpjBnj4fyTxphsY8y/jDH/MMa08ea+IiLBoPV1fbiq7/2BDqNc5XZbNMbUAb4G+gA5QDpwr7U22+2am4DN1tpTxphHAIe1Nr6s+6rboogEC190bPSVsrotejNCvw7Yaa3dZa09CywEBrpfYK1da60tXBf7GRAcnWxEREKINwm9FbDX7Tin4LXSPAh86OmEMSbZGJNhjMk4dOiQ91GKiEi5fPpQ1BgzBIgGpng6b62dZa2NttZGX3rppb58axGRWs+bssV9QITbcXjBa0UYY2KB/wJ6W2vP+CY8ERHxljcj9HSgozGmnTHmQiABWOZ+gTGmO/AGEGetPej7MEVEpDzlJnRr7TlgBLAK2Aa8ba3daoyZZIyJK7hsCnAx8HdjTKYxZlkptxMRET/xaqWotfYD4INir413+z441sWKiFRSVTs2Vgct/RcRKYcvOjZWh3IXFvmLFhaJiFRcVRcWiYhIEFBCFxEJEUroIiIhQgldRCREKKGLiIQIJXQRkRChhC4iEiKU0EVEQkTAFhYZYw4B3wfkzasmDMgNdBB+EsqfDfT5glkofzao2OdrY6312H88YAk9WBljMkpbpRXsQvmzgT5fMAvlzwa++3yachERCRFK6CIiIUIJveJmBToAPwrlzwb6fMEslD8b+OjzaQ5dRCREaIQuIhIilNBFREKEEroXjDERxpi1xphsY8xWY8zjgY7JH4wxdYwx/zTGrAh0LL5mjGlqjFlsjNlujNlmjOkZ6Jh8xRjzRMHfyyxjzN+MMfUDHVNVGGPmGGMOGmOy3F67xBjzkTHmm4J/NgtkjFVRyuebUvB381/GmHeMMU0rc28ldO+cA56y1nYGfgMMN8Z0DnBM/vA4zo3AQ9F0YKW19irgPwiRz2mMaQWMBKKttV2AOkBCYKOqsrlA32KvjQH+Ya3tCPyj4DhYzaXk5/sI6GKt7QZ8DYytzI2V0L1grd1vrf2y4PufcCaD4Nhk0EvGmHCgP/BmoGPxNWNME+BG4C8A1tqz1tqjgY3Kp+oCDYwxdYGGwL8DHE+VWGvXA0eKvTwQmFfw/TzgjmoNyoc8fT5r7Wpr7bmCw8+A8MrcWwm9gowxbYHuwObARuJz04D/BPIDHYgftAMOAakFU0pvGmMuCnRQvmCt3QdMBfYA+4Fj1trVgY3KLy631u4v+P4AcHkgg/GzYcCHlflBJfQKMMZcDCwBRllrjwc6Hl8xxgwADlprvwh0LH5SF7gGeM1a2x04SXD/yu5SMJc8EOf/tH4NXGSMGRLYqPzLOmutQ7Le2hjzXzineP+vMj+vhO4lY0w9nMn8/6y1SwMdj4/1AuKMMd8BC4GbjTHzAxuST+UAOdbawt+qFuNM8KEgFthtrT1krc0DlgLXBzgmf/jBGNMSoOCfBwMcj88ZY4YCA4D7bCUXCCmhe8EYY3DOv26z1v5PoOPxNWvtWGttuLW2Lc4Hah9ba0NmlGetPQDsNcZEFrx0C5AdwJB8aQ/wG2NMw4K/p7cQIg98i1kGPFDw/QPAewGMxeeMMX1xTnnGWWtPVfY+Suje6QXcj3PkmlnwdVugg5IKeQz4P2PMv4Ao4MUAx+MTBb91LAa+BLbg/G86qJfJG2P+BmwCIo0xOcaYB4EUoI8x5hucv5WkBDLGqijl870CNAI+Ksgvr1fq3lr6LyISGjRCFxEJEUroIiIhQgldRCREKKGLiIQIJXQRkRChhC4iEiKU0EVEQsT/B7YjD2mqmcoIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cph.plot_covariate_groups('trt', values=[0, 1]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the group without treatment has a lower survival rate at all times (the x-axis is time) compared to the treatment group."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='6'></a>\n",
    "## 6. Hazard Ratio\n",
    "\n",
    "Recall from the lecture videos that the Hazard Ratio between two patients was the likelihood of one patient (e.g smoker) being more at risk than the other (e.g non-smoker).\n",
    "$$\n",
    "\\frac{\\lambda_{smoker}(t)}{\\lambda_{nonsmoker}(t)} = e^{\\theta (X_{smoker} - X_{nonsmoker})^T}\n",
    "$$\n",
    "\n",
    "Where\n",
    "\n",
    "$$\n",
    "\\lambda_{smoker}(t) = \\lambda_0(t)e^{\\theta X_{smoker}^T}\n",
    "$$\n",
    "and\n",
    "$$\n",
    "\\lambda_{nonsmoker}(t) = \\lambda_0(t)e^{\\theta X_{nonsmoker}^T} \\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-2'></a>\n",
    "### Exercise 2 - hazard_ratio\n",
    "In the cell below, write a function to compute the hazard ratio between two individuals given the cox model's coefficients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>use numpy.dot</li>\n",
    "    <li>use nump.exp</li>\n",
    "</ul>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WbBmxbeDA3k1"
   },
   "outputs": [],
   "source": [
    "# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def hazard_ratio(case_1, case_2, cox_params):\n",
    "    '''\n",
    "    Return the hazard ratio of case_1 : case_2 using\n",
    "    the coefficients of the cox model.\n",
    "    \n",
    "    Args:\n",
    "        case_1 (np.array): (1 x d) array of covariates\n",
    "        case_2 (np.array): (1 x d) array of covariates\n",
    "        model (np.array): (1 x d) array of cox model coefficients\n",
    "    Returns:\n",
    "        hazard_ratio (float): hazard ratio of case_1 : case_2\n",
    "    '''\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    hr = np.exp(cox_params.dot((case_1 - case_2).T))\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return hr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zbDQUxE6CcA3"
   },
   "source": [
    "Now, evaluate it on the following pair of indivduals: `i = 1` and `j = 5`. Given your implementation of `hazard_ratio` is correct, you will **only** pass the test when `i = 1` and `j = 5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7flsvTRXCgqO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.029017732492221 \n",
      "\n",
      "\u001b[92m All tests passed.\n"
     ]
    }
   ],
   "source": [
    "# test cell ex2\n",
    "\n",
    "# Set i = 1 and j = 5 to pass the test\n",
    "i = 1\n",
    "j = 5\n",
    "\n",
    "# DO NOT modify anything below\n",
    "hazard_ratio_test(hazard_ratio, i, j, one_hot_train, cph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected Output:\n",
    "```\n",
    "15.029017732492221 \n",
    "\n",
    " All tests passed.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** \n",
    "\n",
    "Is `case_1` or `case_2` at greater risk? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Check your answer!</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "<ul>\n",
    "    Important! The following answer only applies if you picked i = 1 and j = 5\n",
    "    <li>You should see that `case_1` is at higher risk.</li>\n",
    "    <li>The hazard ratio of case 1 / case 2 is greater than 1, so case 1 had a higher hazard relative to case 2</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect different pairs, and see if you can figure out which patient is more at risk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "g2PZ3sGvCs0K",
    "outputId": "59336868-d421-4645-d88e-76a8a8cffc9f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Case 1\n",
      "\n",
      " trt          0.000000\n",
      "age          0.563645\n",
      "sex          0.000000\n",
      "ascites      0.000000\n",
      "hepato       1.000000\n",
      "spiders      1.000000\n",
      "bili        -0.405645\n",
      "chol        -0.210436\n",
      "albumin      1.514297\n",
      "copper      -0.481961\n",
      "alk.phos     2.173526\n",
      "ast         -0.144699\n",
      "trig        -0.531125\n",
      "platelet    -0.450972\n",
      "protime     -0.139881\n",
      "edema_0.5    0.000000\n",
      "edema_1.0    0.000000\n",
      "stage_2.0    0.000000\n",
      "stage_3.0    1.000000\n",
      "stage_4.0    0.000000\n",
      "Name: 1, dtype: float64 \n",
      "\n",
      "Case 2\n",
      "\n",
      " trt          0.000000\n",
      "age          0.463447\n",
      "sex          0.000000\n",
      "ascites      0.000000\n",
      "hepato       1.000000\n",
      "spiders      0.000000\n",
      "bili        -0.489581\n",
      "chol        -0.309875\n",
      "albumin     -1.232371\n",
      "copper      -0.504348\n",
      "alk.phos     2.870427\n",
      "ast         -0.936261\n",
      "trig        -0.150229\n",
      "platelet     3.190823\n",
      "protime     -0.139881\n",
      "edema_0.5    0.000000\n",
      "edema_1.0    0.000000\n",
      "stage_2.0    0.000000\n",
      "stage_3.0    0.000000\n",
      "stage_4.0    1.000000\n",
      "Name: 38, dtype: float64 \n",
      "\n",
      "Hazard Ratio: 0.1780450006997129\n"
     ]
    }
   ],
   "source": [
    "i = 4\n",
    "case_1 = one_hot_train.iloc[i, :].drop(['time', 'status'])\n",
    "\n",
    "j = 7\n",
    "case_2 = one_hot_train.iloc[j, :].drop(['time', 'status'])\n",
    "\n",
    "print(\"Case 1\\n\\n\", case_1, \"\\n\")\n",
    "print(\"Case 2\\n\\n\", case_2, \"\\n\")\n",
    "print(\"Hazard Ratio:\", hazard_ratio(case_1.values, case_2.values, cph.params_.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Check your answer!</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "<ul>\n",
    "    Important! The following answer only applies if you picked i = 4 and j = 7\n",
    "    <li>You should see that `case_2` is at higher risk.</li>\n",
    "    <li>The hazard ratio of case 1 / case 2 is less than 1, so case 2 had a higher hazard relative to case 1</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KUa6r-KOyySp"
   },
   "source": [
    "<a name='7'></a>\n",
    "## 7. Harrell's C-index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "woQAtSmRXrgr"
   },
   "source": [
    "To evaluate how good our model is performing, we will write our own version of the C-index. Similar to the week 1 case, C-index in the survival context is the probability that, given a randomly selected pair of individuals, the one who died sooner has a higher risk score. \n",
    "\n",
    "However, we need to take into account censoring. Imagine a pair of patients, $A$ and $B$. \n",
    "\n",
    "#### Scenario 1\n",
    "- A was censored at time $t_A$ \n",
    "- B died at $t_B$\n",
    "- $t_A < t_B$. \n",
    "\n",
    "Because of censoring, we can't say whether $A$ or $B$ should have a higher risk score. \n",
    "\n",
    "#### Scenario 2\n",
    "Now imagine that $t_A > t_B$.\n",
    "\n",
    "- A was censored at time $t_A$ \n",
    "- B died at $t_B$\n",
    "- $t_A > t_B$\n",
    "\n",
    "Now we can definitively say that $B$ should have a higher risk score than $A$, since we know for a fact that $A$ lived longer. \n",
    "\n",
    "Therefore, when we compute our C-index\n",
    "- We should only consider pairs where at most one person is censored\n",
    "- If they are censored, then their censored time should occur *after* the other person's time of death. \n",
    "\n",
    "The metric we get if we use this rule is called **Harrel's C-index**.\n",
    "\n",
    "Note that in this case, being censored at time $t$ means that the true death time was some time AFTER time $t$ and not at $t$. \n",
    "- Therefore if $t_A = t_B$ and A was censored:\n",
    "    - Then $A$ actually lived longer than $B$. \n",
    "    - This will effect how you deal with ties in the exercise below!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-3'></a>\n",
    "### Exercise 3 - harrell_c\n",
    "Fill in the function below to compute Harrel's C-index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>If you get a division by zero error, consider checking how you count when a pair is permissible (in the case where one patient is censored and the other is not censored).</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def harrell_c(y_true, scores, event):\n",
    "    '''\n",
    "    Compute Harrel C-index given true event/censoring times,\n",
    "    model output, and event indicators.\n",
    "    \n",
    "    Args:\n",
    "        y_true (array): array of true event times\n",
    "        scores (array): model risk scores\n",
    "        event (array): indicator, 1 if event occurred at that index, 0 for censorship\n",
    "    Returns:\n",
    "        result (float): C-index metric\n",
    "    '''\n",
    "    \n",
    "    n = len(y_true)\n",
    "    assert (len(scores) == n and len(event) == n)\n",
    "    \n",
    "    concordant = 0.0\n",
    "    permissible = 0.0\n",
    "    ties = 0.0\n",
    "    \n",
    "    result = 0.0\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' and 'pass' with your code) ###\n",
    "    \n",
    "  # use double for loop to go through cases\n",
    "    for i in range(n):\n",
    "        # set lower bound on j to avoid double counting\n",
    "        for j in range(i+1, n):\n",
    "            \n",
    "            # check if at most one is censored\n",
    "            if event[i] == 1 or event[j] == 1:\n",
    "                # check if neither are censored\n",
    "                if event[i] == 1 and event[j] == 1:\n",
    "                    \n",
    "                    permissible += 1.0\n",
    "                    \n",
    "                    # check if scores are tied\n",
    "                    if scores[i] == scores[j]:\n",
    "                        ties += 1.0\n",
    "                    # check for concordant\n",
    "                    elif y_true[i] < y_true[j] and scores[i] > scores[j]:\n",
    "                        concordant += 1.0\n",
    "                    elif y_true[i] > y_true[j] and scores[i] < scores[j]:\n",
    "                        concordant += 1.0\n",
    "                \n",
    "                # check if one is censored\n",
    "                elif event[i] != event[j] :   \n",
    "                    # get censored index\n",
    "                    censored = j\n",
    "                    uncensored = i\n",
    "                    \n",
    "                    if event[i] == 0:\n",
    "                        censored = i\n",
    "                        uncensored = j\n",
    "                        \n",
    "                    # check if permissible\n",
    "                    # Note: in this case, we are assuming that censored at a time\n",
    "                    # means that you did NOT die at that time. That is, if you\n",
    "                    # live until time 30 and have event = 0, then you lived THROUGH\n",
    "                    # time 30.\n",
    "                    if y_true[uncensored] <= y_true[censored]:\n",
    "                        permissible += 1.0\n",
    "                        \n",
    "                        # check if scores are tied\n",
    "                        if scores[uncensored] == scores[censored]:\n",
    "                            # update ties \n",
    "                            ties += 1.0\n",
    "                            \n",
    "                        # check if scores are concordant \n",
    "                        if scores[uncensored] > scores[censored]:\n",
    "                            concordant += 1.0\n",
    "    \n",
    "    # set result to c-index computed from number of concordant pairs,\n",
    "    # number of ties, and number of permissible pairs (REPLACE 0 with your code)  \n",
    "    result = (concordant + 0.5*ties) / permissible\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return result   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Case 1\n",
      "Output: 1.0\n",
      "\n",
      "Test Case 2\n",
      "Output: 0.0\n",
      "\n",
      "Test Case 3\n",
      "Output: 1.0\n",
      "\n",
      "Test Case 4\n",
      "Output: 0.75\n",
      "\n",
      "Test Case 5\n",
      "Output: 0.5833333333333334\n",
      "\n",
      "Test Case 6\n",
      "Output: 1.0\n"
     ]
    }
   ],
   "source": [
    "# test cell ex1 - do not modify this test cell\n",
    "y_true = [30, 12, 84, 9]\n",
    "\n",
    "# Case 1\n",
    "event = [1, 1, 1, 1]\n",
    "scores = [0.5, 0.9, 0.1, 1.0]\n",
    "print(\"Test Case 1\")\n",
    "print(\"Output: {}\".format(harrell_c(y_true, scores, event)))\n",
    "\n",
    "# Case 2\n",
    "scores = [0.9, 0.5, 1.0, 0.1]\n",
    "print(\"\\nTest Case 2\")\n",
    "print(\"Output: {}\".format(harrell_c(y_true, scores, event)))\n",
    "\n",
    "# Case 3\n",
    "event = [1, 0, 1, 1]\n",
    "scores = [0.5, 0.9, 0.1, 1.0]\n",
    "print(\"\\nTest Case 3\")\n",
    "print(\"Output: {}\".format(harrell_c(y_true, scores, event)))\n",
    "\n",
    "# Case 4\n",
    "y_true = [30, 30, 20, 20]\n",
    "event = [1, 0, 1, 0]\n",
    "scores = [10, 5, 15, 20]\n",
    "print(\"\\nTest Case 4\")\n",
    "print(\"Output: {}\".format(harrell_c(y_true, scores, event)))\n",
    "\n",
    "# Case 5\n",
    "y_true = list(reversed([30, 30, 30, 20, 20]))\n",
    "event = [0, 1, 0, 1, 0]\n",
    "scores = list(reversed([15, 10, 5, 15, 20]))\n",
    "print(\"\\nTest Case 5\")\n",
    "print(\"Output: {}\".format(harrell_c(y_true, scores, event)))\n",
    "\n",
    "# Case 6\n",
    "y_true = [10,10]\n",
    "event = [0,1]\n",
    "scores = [4,5]\n",
    "print(\"\\nTest Case 6\")\n",
    "print(\"Output: {}\".format(harrell_c(y_true, scores, event)))  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Expected Output:\n",
    "```\n",
    "Test Case 1\n",
    "Output:  1.0\n",
    "\n",
    "Test Case 2\n",
    "Output:  0.0\n",
    "\n",
    "Test Case 3\n",
    "Output:  1.0\n",
    "\n",
    "Test Case 4\n",
    "Output:  0.75\n",
    "\n",
    "Test Case 5\n",
    "Output:  0.5833333333333334\n",
    "\n",
    "Tes Case 6\n",
    "Output:  1.0 \n",
    "\n",
    " All tests passed.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CtQVe4pAn8ic"
   },
   "source": [
    "Now use the Harrell's C-index function to evaluate the cox model on our data sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "8nzHc_Qbn7dM",
    "outputId": "bc2f960d-16e5-46b2-a41f-695892c311c7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 0.8265139116202946\n",
      "Val: 0.8544776119402985\n",
      "Test: 0.8478543563068921\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "scores = cph.predict_partial_hazard(one_hot_train)\n",
    "cox_train_scores = harrell_c(one_hot_train['time'].values, scores.values, one_hot_train['status'].values)\n",
    "# Validation\n",
    "scores = cph.predict_partial_hazard(one_hot_val)\n",
    "cox_val_scores = harrell_c(one_hot_val['time'].values, scores.values, one_hot_val['status'].values)\n",
    "# Test\n",
    "scores = cph.predict_partial_hazard(one_hot_test)\n",
    "cox_test_scores = harrell_c(one_hot_test['time'].values, scores.values, one_hot_test['status'].values)\n",
    "\n",
    "print(\"Train:\", cox_train_scores)\n",
    "print(\"Val:\", cox_val_scores)\n",
    "print(\"Test:\", cox_test_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do these values tell us ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AuNjR_wNkpWz"
   },
   "source": [
    "<a name='8'></a>\n",
    "## 8. Random Survival Forests\n",
    "\n",
    "This performed well, but you have a hunch you can squeeze out better performance by using a machine learning approach. You decide to use a Random Survival Forest. To do this, you can use the `RandomForestSRC` package in R. To call R function from Python, we'll use the `r2py` package. Run the following cell to import the necessary requirements. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 530
    },
    "colab_type": "code",
    "id": "ZgSy-Dj6kquK",
    "outputId": "4aa5d2fa-30f4-4328-ae29-a2ff05223e22"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: Loading required package: ggplot2\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext rpy2.ipython\n",
    "%R require(ggplot2)\n",
    "\n",
    "from rpy2.robjects.packages import importr\n",
    "# import R's \"base\" package\n",
    "base = importr('base')\n",
    "\n",
    "# import R's \"utils\" package\n",
    "utils = importr('utils')\n",
    "\n",
    "# import rpy2's package module\n",
    "import rpy2.robjects.packages as rpackages\n",
    "\n",
    "forest = rpackages.importr('randomForestSRC', lib_loc='R')\n",
    "\n",
    "from rpy2 import robjects as ro\n",
    "R = ro.r\n",
    "\n",
    "from rpy2.robjects import pandas2ri\n",
    "pandas2ri.activate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LXBgqQBfuMA5"
   },
   "source": [
    "Instead of encoding our categories as binary features, we can use the original dataframe since trees deal well with raw categorical data (can you think why this might be?).\n",
    "\n",
    "Run the code cell below to build your forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B-pio4o4mdVJ"
   },
   "outputs": [],
   "source": [
    "model = forest.rfsrc(ro.Formula('Surv(time, status) ~ .'), data=df_train, ntree=300, nodedepth=5, seed=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "id": "zZfcUvJ3nL04",
    "outputId": "27d00bd8-ea33-4c1b-f5d7-ca2c73ead721"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                         Sample size: 154\n",
      "                    Number of deaths: 64\n",
      "                     Number of trees: 300\n",
      "           Forest terminal node size: 15\n",
      "       Average no. of terminal nodes: 6.54\n",
      "No. of variables tried at each split: 5\n",
      "              Total no. of variables: 17\n",
      "       Resampling used to grow trees: swor\n",
      "    Resample size used to grow trees: 97\n",
      "                            Analysis: RSF\n",
      "                              Family: surv\n",
      "                      Splitting rule: logrank *random*\n",
      "       Number of random split points: 10\n",
      "                          Error rate: 19.07%\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9Mwzm55H-QKV"
   },
   "source": [
    "Finally, let's evaluate on our validation and test sets, and compare it with our Cox model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "vfl4LbGfpbKp",
    "outputId": "13f8b560-e171-41e9-f6dc-cf0468a3f786"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cox Model Validation Score: 0.8544776119402985\n",
      "Survival Forest Validation Score: 0.8296019900497512\n"
     ]
    }
   ],
   "source": [
    "result = R.predict(model, newdata=df_val)\n",
    "scores = np.array(result.rx('predicted')[0])\n",
    "\n",
    "print(\"Cox Model Validation Score:\", cox_val_scores)\n",
    "print(\"Survival Forest Validation Score:\", harrell_c(df_val['time'].values, scores, df_val['status'].values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "uhqSQJhrplSG",
    "outputId": "752c266e-0234-45c5-d53f-554e2ff17a5a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cox Model Test Score: 0.8478543563068921\n",
      "Survival Forest Validation Score: 0.8621586475942783\n"
     ]
    }
   ],
   "source": [
    "result = R.predict(model, newdata=df_test)\n",
    "scores = np.array(result.rx('predicted')[0])\n",
    "\n",
    "print(\"Cox Model Test Score:\", cox_test_scores)\n",
    "print(\"Survival Forest Validation Score:\", harrell_c(df_test['time'].values, scores, df_test['status'].values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gp_SgUXreAWn"
   },
   "source": [
    "Your random forest model should be outperforming the Cox model slightly. Let's dig deeper to see how they differ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZtPMPaSli8GB"
   },
   "source": [
    "<a name='9'></a>\n",
    "## 9. Permutation Method for Interpretation\n",
    "\n",
    "We'll dig a bit deeper into interpretation methods for forests a bit later, but for now just know that random surival forests come with their own built in variable importance feature. The method is referred to as VIMP, and for the purpose of this section you should just know that higher absolute value of the VIMP means that the variable generally has a larger effect on the model outcome.\n",
    "\n",
    "Run the next cell to compute and plot VIMP for the random survival forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "colab_type": "code",
    "id": "u7M4_N_d-YJu",
    "outputId": "7e1830cb-4b67-444f-8ba5-d49d3ff2f172"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEICAYAAABI7RO5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZxcZZn28d+VgISwJECiAoItCLITsUFZVER0EBxFQFFARNQMgvDiO6g4MiM4LijvjBvjEhlkdSMM6IACKouCYkhIyEIMKokiKBKWsIQlJNf7x3laiqI66equ6qpOru/n058+dc5znnNXEfqu5zzn3Ee2iYiIGKhRnQ4gIiJGliSOiIhoShJHREQ0JYkjIiKaksQRERFNSeKIiIimJHHEGk/S5ySdPMC210t6f4uPf7qki1rZZztIWiRp/xb0c6Kkz7cipuiMJI7oCElXSfpUg/VvlfRXSWtJOk/Sp8v6HkmWNLOu/QRJT0laVLNukaTHJT0q6d7Sz/r9xDEROBr4ZkvfYJuMlCSzCt8CjpT0/E4HEoOTxBGdcj5wlCTVrX83cLHtp/vZb6yknWpeHwEsbNDuH22vD+wG9AKn9dPfMcCPbT8+4MhjSGw/AfyEKmHHCJTEEZ1yObAJ8Oq+FZI2At4MXLCS/S4E3lPz+uiVtbd9N9UfqZ36afIm4IbaGCRdIek+SQ+W5RfV7bO1pGmSHpb0Q0kbl33HSLpI0v2SHpJ0i6QXlG2bSfqRpAck/V7SBxoFI2lfSX+uW7dI0v6SDgD+BTi8jKZuK9vHSfpvSX+RdLekT0sa3aDvzcpIbOOadS+XtFjS2pK2lnRtiX+xpIslje8nzr+PBhvFXY51afkcF0o6qa6L64GDGvUd3S+JIzqifMP/Ac/+1vkO4Le2b1vJrhcB75Q0WtIOwPrAb/prLGkL4EBgZj9NdgYW1LweBXwbeDGwJfA4cHbdPkcDxwKbAk8DXynr3wOMA7agSorHlf0Bvgf8GdgMOAz4rKT9VvI+n8P2VcBnge/bXt/2rmXTeSWOlwIvB94IPGcexvY9wK+BQ2tWHwFMtb0MEPC5EuP25X2c3kyMAJJGAf8L3AZsDrweOFnSP9Q0mw/s2mD3GAGSOKKTzgcOkzSmvD66rFuZP1P9od+/tL+wn3aXS3oIuJFqRPHZftqNBx7pe2H7ftuX2l5q+xHgM8Br6/a50PZc248B/wq8o3zDX0aVMF5qe7ntGbYfLslrb+Bjtp+wPQs4hxacqikjmgOBk20/ZvtvwBeBd/azy3eAd5V9Vdp9p7z339v+qe0nbd8H/GeD9z4QuwMTbX/K9lO276Sa16iN6RGqJBsj0FqdDiDWXLZvlLQYOFjSLcAewCED2PUCqrmJvahOdW3boM3Btn82gL4eBDboeyFpLNUf3gOAjcrqDSSNtr28vL6rZv8/AmsDE6iS2BbA98opnouAT1B9g3+gJKLa/XoHEN+qvLgc/y8100Wj6mKsdSnwVUmbUn1uK4Bfwt+T0JepPtMNSj8PDjKmzUri7jO67zjFBsCSQfQdXSAjjui0C6i+eR8FXG373gHscynV+fE7bf9piMefzbMTzz8DLwNeaXtD4DVlfe0k/hY1y1tSjTQW215m+wzbO1AltTdTvbd7gI0lbVC3390N4nkMGNv3ooxkJtZsry9nfRfwJDDB9vjys6HtHRu9WdsPAtcAh1OdpvqenymR/dnS/87lvR9V9777jRN4YV1MC2viGW97A9sH1rTZnupUVoxASRzRaRdQnXb6AKs+TQVAOUW0Hw3O4w/Cj3n26ZgNqOYlHiqTyJ9ssM9RknYoo5NPUc0RLJf0Okk7lz/2D1MllBW27wJ+BXyuTKDvAryPakRS7w5gjKSDJK1NdTXYOjXb7wV6yjwCtv9ClQj+Q9KGkkaVSe6VnWL6DlVCO6ws1773R4ElkjYHPrKSPmYBB0raWNILgdr7YKYBj0j6mKR1y3zUTpJ2r2nzWqqLFmIESuKIjrK9iOqP6nrAj5rYb7rtP7QghAuo/gCuW15/CVgXWAzcDFzVYJ8LqSak/wqMAfquGHohMJUqacynmlvpm4N5F9BDNfq4DPhko1NptpcAx1PNgdxN9c2+9iqrS8rv+yXdWpaPBp4H3E51amkq1cR9f34EbAP8te5ChDOoLl9eAlwJ/M9K+riQasSwiCpxfb/mPSynGm1NorpUenF5P+OguvqMal5mQF8UovsoD3KKNZ2kzwJ/s/2lTseyJpB0IrCF7Y92OpYYnCSOiIhoSk5VRUREU5I4IiKiKUkcERHRlNX+BsAJEya4p6en02FERIwoM2bMWGx7YqNtq33i6OnpYfr06Z0OIyJiRJH0x/625VRVREQ0JYkjIiKaksQRERFNSeKIiIimJHFERERTkjgiIqIpSRwREdGUJI6IiGjKan8D4Jy7l9Bz6pXPWb/ozIM6EE1ExMiXEUdERDRlWBOHpJPL4zYjImKEGu4Rx8k8+wH3ERExwrQtcUhaT9KVkm6TNFfSJ4HNgOskXVfafF3SdEnzJJ1Rs++Bkn4raYakr0i6oqbPcyVNkzRT0lvbFX9ERDTWzsnxA4B7bB8EIGkc8F7gdbYXlzafsP2ApNHAzyXtAtwBfBN4je2Fkr5b0+cngGttHytpPDBN0s9sP1Z7YEmTgckAozdsWBU4IiIGqZ2nquYAb5D0eUmvtr2kQZt3SLoVmAnsCOwAbAfcaXthaVObON4InCppFnA9MAbYsr5T21Ns99ruHT12XOveUUREtG/EYfsOSbsBBwKflvTz2u2SXgKcAuxu+0FJ51ElgpURcKjtBe2IOSIiVq2dcxybAUttXwScBewGPAJsUJpsCDwGLJH0AuBNZf0CYCtJPeX14TXdXg2cKEnlGC9vV/wREdFYO+c4dgbOkrQCWAZ8ENgTuErSPbZfJ2km8FvgLuAmANuPSzq+tHsMuKWmz38HvgTMljQKWAi8uY3vISIi6sh2p2N4Dknr2360jCz+C/id7S8Opq/e3l7n0bEREc2RNMN2b6Nt3Xrn+AfKBPg8YBzVVVYREdEFunLE0UrrbLqNN33Plzpy7NTDioiRaiSOOCIioksNd62qY8rVVn2vz5G0w3DGEBERQ9PyxFHuAu/PMVRlRwCw/X7bt7c6hoiIaJ+mEoeknlJD6mJJ8yVNlTRW0qJyh/itwNslTZJ0s6TZki6TtJGkw4Be4GJJsyStK+l6Sb2l70clnVXqVv1M0h5l+52S3lLajC5tbil9/1PLP5GIiFipwYw4XgZ8zfb2wMPA8WX9/bZ3s/094ALgY7Z3oSo98knbU4HpwJG2J9l+vK7f9ajqUO1IdaPgp4E3AG8DPlXavA9YYnt3YHeqq69eUh+gpMmleOL05UsbVTqJiIjBGkziuMv2TWX5ImCfsvx9+Hsxw/G2byjrzwdeM4B+nwKuKstzgBtsLyvLPWX9G4Gjy6W6vwE2Abap7yi1qiIi2mcwd47XX7/b9/qx+oZNWuZnrg1eATwJYHuFpL44BZxo++ohHisiIgZpMCOOLSXtWZaPAG6s3Viq4D4o6dVl1buBvtFHba2qwbga+KCktQEkbStpvSH0FxERTRrMiGMBcIKkc4Hbga8DJ9a1eQ/wjfKY2DupnsMBcF5Z/zhV3apmnUN12urWUo7kPuDgle2w8+bjmJ4b8SIiWqapO8dLxdorbO/UroBaLbWqIiKalzvHIyKiZZo6VWV7ETBiRhsAc+5eQs+pVz5rXWpIRUQMXstGHLU3862kzcll3mPY+oqIiNYa7lNVJwOt+mPfyr4iImKAmk4c/ZUdqWvz9XLn9jxJZ5R1J1HVqbpO0nVl3Rsl/VrSrZIukbR+g+M9p02jviIiYngMdsTRX9mRPp8os/G7AK+VtIvtrwD3AK8rj42dAJwG7G97N6pyJP+3tpP+2tT3VR9cSo5ERLTPYJ85Xl925KS67e+QNLn0vymwAzC7rs2ryvqbqlsyeB7w60G0eQ7bU4ApUD3IaWBvKSIiBmKwiaO/siOUooOnALvbflDSecCYBn0I+Kntd63kOANpExERw2iwp6pWVnZkQ6q6VUskvQB4U8222pIjNwN7S3opgKT1JG1bd5yVtRlq+ZKIiBiEwSaOvrIj84GNqMqOAGD7NmAm8FvgO8BNNftNAa6SdJ3t+6ge7PRdSbOpTkFtV3uQVbT5e1+DfA8RETEITZUcgZFXdiQlRyIimpeSIxER0TJNT46PxLIjERHROqv9iKNRraqIiBi8jicOSeMl1d9AWLv9V8MZT0RErFzHEwcwnufeeU7f42Jt7zXsEUVERL8GewNgK50JbC1pFrAMeAJ4kOqy220lPWp7fUmjgLOB/YC7SttzbU/tUNwREWukbkgcpwI72Z4kaV/gyvJ6YV27Q6geG7sD8HxgPnBuow5LuZPJAKM3nNieqCMi1lDdcKqq3rQGSQNgH+AS2yts/xXo98Y/21Ns99ruHT12XNsCjYhYE3Vj4nis0wFERET/uiFxDLTm1E3AoZJGlRpY+7Y1qoiIaKjjcxy275d0k6S5wOPAvf00vRR4PXA71eT4rUAethERMcyarlXVSZLWt/2opE2AacDeZb6jX6lVFRHRvJXVqur4iKNJV0gaT/VAp39fVdKIiIjWG1GJw/a+ze4zkJIji848aLAhRUSscbphcjwiIkaQEZk4JB0jabNOxxERsSYakYmD6qmASRwRER3QVXMcki4HtgDGAF8G/rv89AKmKjFyV3l9saTHgT1tP96ZiCMi1jxdlTiAY20/IGld4BZgBrB532NqJY23/ZCkDwGn2G54nW1qVUVEtE+3nao6SdJtwM1UI4/nAVtJ+qqkA4CHB9JJalVFRLRP1ySOUhl3f6pTT7sCM4F1gF2B64HjgHM6FV9ERFS66VTVOOBB20slbQe8CpgAjLJ9qaQFwEWl7UDrW0VERIt1U+K4CjhO0nxgAdXpqs2B68tDnAA+Xn6fB3wjk+MREcNvRNWqGozUqoqIaN7KalV1zRxHRESMDKt94uirVbWqelURETEwq33iiIiI1hp04pC0SNIEST3lIUxD6mew+0dExPDKiCMiIpoyoMQh6XJJMyTNK+U8+mu3laSZknavW98j6beSLpY0X9JUSWNrmpwo6VZJc8o9HEjauBx3tqSbJe1S1r9W0qzyM1NS7ueIiBhGAx1xHGv7FVTFBU8qj259Fkkvo3ou+DG2b2nQx8uAr9nenqp0yPE12xbb3g34OnBKWXcGMNP2LsC/ABeU9acAJ9ieBLya6jnl9bFMljRd0vTlS/NY8oiIVhpo4qivIbVN3faJwA+BI23f1k8fd9m+qSxfBOxTs+1/yu8ZQE9Z3ge4EMD2tcAmkjYEbgL+U9JJwHjbT9cfKLWqIiLaZ5WJo58aUmPqmi0B/sSzk0G9+jsNa18/WX4vZxV3s9s+E3g/sC5wU9+prYiIGB4DGXE0qiFV7yngbcDRko7op58tJe1Zlo8AblzFcX8JHAl/T16LbT8saWvbc2x/nqr0ehJHRMQwGkitqkY1pJ7D9mOS3gz8VNKjwHTgHNsHliYLgBMknQvcTjWfsTKnA+dKmg0sBd5T1p8s6XXACmAe8JOVdbLz5uOYfuZBqzhUREQM1LDUqpLUA1zR90Cm4ZRaVRERzUutqoiIaJlhKatuexEw7KMNeKZWVSstyqmviFiDZcQRERFNGfGJQ9LoTscQEbEmGXLikHR0KQtym6QLS3mRa8u6n0vasrQ7T9I3yh3dd5QrsJB0jKQfSrpe0u8kfbKm76MkTSvlRb7ZlyQkPSrpP8pNiXs2DCwiItpiSHMcknYETgP2sr1Y0sbA+cD5ts+XdCzwFeDgsksPsAewNXCdpJeW9XtQzYEsBW6RdCXwGHA4sLftZZK+RnVfxwXAesBvbP/zUOKPiIjmDXVyfD/gEtuLAWw/UG7yO6RsvxD4Qk37H9heAfxO0p08c/PeT23fDyDpf6juQH8aeAVVIoHqTvG/lfbLqepiNVQKMU4GGL3hxCG+xYiIqDUsV1XV6K/sSKP1ohq5fLxBP0/YXt7vQewpwBSAdTbdZvV+qHpExDAb6hzHtcDb+6rlllNVvwLeWbYfSVU6pM/bJY2StDWwFdXd5ABvKGXU16U6rXUT8HPgMEnP7+tb0ouHGG9ERAzRkEYctudJ+gxwg6TlVAUQTwS+LekjwH3Ae2t2+RMwDdgQOM72E+U01DSqU08vAi6yPR1A0mnANZJGAcuAE4A/DiXmiIgYmmEpOQLVVVVUZUem1q0/Bui1/aF2HDclRyIimpeSIxER0TLDNjlu+5h+1p8HnDdccURExNAM91VVw64dtapqpW5VRKxphnSqStIiSRPK3eJzWxVU3THOkbRDO/qOiIjmdf2Iw/b7Ox1DREQ8Y8AjDkmXS5ohaV65M7veWpIuljRf0lRJY8t+iyRNKMu9kq4vy6dLOl/SLyX9UdIhkr4gaY6kqyStXdpdL6m3LD8q6TOlLtbNkl4w1A8gIiKa08ypqmNtvwLoBU7qu+mvxsuAr9neHngYOH4AfW5NVbbkLcBFwHW2dwYeBxpNHqwH3Gx7V+AXwAcadSppcimmOH350iUDCCMiIgaqmcRxUqlGezOwBbBN3fa7bN9Uli+iqje1Kj+xvQyYA4ymer455XVPg/ZPAVeU5Rn9tMH2FNu9tntHjx03gDAiImKgBjTHIWlfYH9gT9tLy+mmMXXN+qtD9TTPJKj6fZ4EsL1C0jI/czfiin5iq22zfKDxR0RE6wx0xDEOeLAkje2AVzVos2WpjAtwBHBjWV5EVeUW4NDBBhoREd1hoInjKqrJ7/nAmVSnq+otAE4obTYCvl7WnwF8WdJ0qlFCRESMYMNWq6pTUqsqIqJ5qVUVEREts9pPLjdTciTlQyIiVi0jjoiIaErXJA5J50k6rIn2bauPFRER/euaxBERESNDxxKHpKMlzS51py4sq18j6VeS7uwbfahylqS5pY7V4Z2KOSIiOjQ5LmlH4DRgL9uLJW0M/CewKVWpku2AHwFTgUOAScCuwATgFkm/WEX/k4HJAKM3nNiutxERsUbq1IhjP+AS24sBbD9Q1l9ue4Xt24G+yrf7AN+1vdz2vcANwO4r6zy1qiIi2qfb5jierFlWx6KIiIh+dSpxXAu8va80ezlV1Z9fAodLGi1pIvAaYNowxBgREQ10ZI7D9jxJnwFukLQcmLmS5pcBewK3UVXc/ajtv0rqaXugERHxHKlVFRERz5FaVRER0TKpVRURsRpqZ+29jDgiIqIpXZU4+qs/JekcSTuU5UWSJpTlXw13jBERa7oRcarK9vv7Wb/XcMcSEbGm66oRR7GWpIslzZc0VdJYSddLes7svqRHOxFgRMSarBsTx8uAr9neHngYOL7ZDiRNljRd0vTlS5e0PMCIiDVZNyaOu2zfVJYvoqpV1ZTUqoqIaJ9uTBz1dySu3ncoRkSMMN2YOLaUtGdZPgK4sZPBRETEs3Vj4lgAnCBpPrAR8PUOxxMRETVSqyoiIp4jtaoiIqJlRsQNgEORWlXRp521eyLWJG0fcUg6RtLZ7T5OREQMj5yqioiIpgw5cUg6StI0SbMkfbM84vW9ku6QNA3Yu6btREmXSrql/Oxd1p8u6XxJv5T0R0mHSPqCpDmSrpK0dmn3b2W/uZKmSMpzySMihtmQEoek7YHDgb1tTwKWA0cBZ1AljH2AHWp2+TLwRdu7A4cC59Rs2xrYD3gL1R3j19neGXgc6Ds5fbbt3W3vBKwLvHko8UdERPOGOjn+euAVwC3ly/+6wF7A9bbvA5D0fWDb0n5/YIeagcKGktYvyz+xvUzSHGA0cFVZPwfoKcuvk/RRYCywMTAP+N/6oCRNBiYDjN5w4hDfYkRE1Bpq4hBwvu2P/32FdDBwSD/tRwGvsv3EszqpEsmTALZXSFrmZ24wWUFVMXcM8DWg1/Zdkk4HxjQ6iO0pwBSAdTbdZvW+USUiYpgNdY7j58Bhkp4PIGljYCbwWkmblLmJt9e0vwY4se+FpElNHKsvSSwuo5TDhhR5REQMypBGHLZvl3QacI2kUcAy4ATgdODXwEPArJpdTgL+S9LscuxfAMcN8FgPSfoWMBf4K3DLUGKPiIjBScmRiIh4jpQciYiIlkniiIiIpqxxtapSrygiYmiGo1bVcZKObrC+R9Lcdh8/IiJaq+0jDtvfaEU/ktay/XQr+oqIiMEb1IhD0nqSrpR0W6kbdbikRTX1paZJemlpe7qkU8ryK8o+t1FdttvX32hJZ5U6VLMl/VNZv2+pX/Uj4PZGxx36RxAREc0Y7KmqA4B7bO9a6kb1lQdZUupLnQ18qcF+3wZOtL1r3fr3lX13B3YHPiDpJWXbbsD/sb3tSo77LJImS5ouafrypUsG+RYjIqKRwSaOOcAbJH1e0qtt9/11/m7N7z1rd5A0Hhhv+xdl1YU1m98IHC1pFvAbYBNgm7Jtmu2Fqzjus9ieYrvXdu/oseMG+RYjIqKRQSUO23dQjQTmAJ+W9G99m2qbNdGlqEYik8rPS2xfU7Y9NoDjRkTEMBnsHMdmwFLbFwFnUf0xh6rEet/vX9fuY/sh4CFJ+5RVR9Zsvhr4YM1zN7aVtF4Tx42IiGEy2KuqdgbOkrSCqj7VB4GpwEalDtWTwLsa7Pde4FxJpip42OccqtLpt5aHM90HHDzA40ZExDBqWa0qSYuoSp4vbkmHLZJaVRERzUutqoiIaJmW3QBou6dVfbXSnLtzOW5ERCtlxBEREU0ZUuJod72p0v8R7eo/IiKa1+0jjh4giSMioou0InGMlvQtSfMkXSNpXUlbS7pK0oxSa2o7AEnnSfpGKQdyh6Q3l/U9pd2t5Wev0veZwKslzZL0YUljJH271MOaKel1LYg/IiKa0IrJ8W2Ad9n+gKQfAIdS3a9xnO3fSXol8DVgv9K+B9gD2Bq4rhRD/BvwBttPSNqGqmRJL3AqcIrtvgTzz4Bt71yS0TWStrX9RG1AkiYDkwFGbzixBW8xIiL6tCJxLLQ9qyzPoEoMewGXVPfyAbBOTfsf2F4B/E7SncB2wELgbEmTgOXAtv0cax/gqwC2fyvpj6Xt7NpGtqcAUwDW2XSb1fuh6hERw6wViePJmuXlwAuAh2xP6qd9/R9yAx8G7gV2pTp99kT9ThER0R3aMTn+MLBQ0tsBVKkto/52SaMkbQ1sBSwAxgF/KSORdwOjS9tHgA1q9v0lpcaVpG2BLcv+ERExTNp1VdWRwPvKA5vmAW+t2fYnYBrwE6p5kCeo5kDeU9pvxzMVcWcDy8uDmz5c2o2SNAf4PnCM7doRT0REtFnLalUN6GDSecAVtqcO1zFTqyoionmpVRURES3TslpVA2H7mOE8HlS1qnpOvXLQ+y8686AWRhMRMfJlxBEREU3paOKQtJmkqWV5kqQDOxlPRESsWkcTh+17bB9WXk4CkjgiIrrckBOHpMtLTap5kiZLGl1qUs0tNaU+XNq9VNLPyqW1t5Z6Vj2l3fOATwGHl7pUh0taT9K5kqaVulRvLf3sWNbNkjS7lCiJiIhh0orJ8WNtPyBpXeAWqrIjm9veCUDS+NLuYuBM25dJGkOVtJ4PYPspSf9G9ejZD5X9Pgtca/vY0sc0ST8DjgO+bPviknBGUye1qiIi2qcVp6pOKjfu3QxsATwP2ErSVyUdADwsaQOqZHIZgO0nbC9dRb9vBE6VNAu4HhhDdaf4r4F/kfQx4MW2H6/f0fYU2722e0ePHdeCtxgREX2G+iCnfYH9gT1t7wrMpCpouCvVH/vjgHMG2z1wqO1J5WdL2/Ntfwd4C/A48GNJ+628m4iIaKWhjjjGAQ/aXlrKnL8KmACMsn0pcBqwm+1HgD9LOhhA0jqSxtb1VV+X6mrgRJUSu5JeXn5vBdxp+yvAD4FdhvgeIiKiCUNNHFcBa0maT/XQpZuBzYHryymmi4CPl7bvpjqtNRv4FfDCur6uA3bomxwH/h1YG5gtaV55DfAOYG7pfyfggiG+h4iIaMKw1qrqhNSqiohoXmpVRUREy6z2iWPO3Us6HUJExGpltU8cERHRWkkcERHRlCSOiIhoSscTR6lJdWWpYTW31Kl6haQbSg2sqyVtKmktSbeUmw6R9DlJn+lw+BERa5xhfZBTPw4A7rF9EICkcVTPI3+r7fvKPR2fKTWrjgGmSjqx7PfKRh2mVlVERPt0Q+KYA/yHpM8DVwAPUt3Y99Ny0/ho4C8AtudJurC029P2U406tD0FmAKwzqbbrN43qkREDLOOJw7bd0jajepZHJ8GrgXm2d6zn112Bh6iVNaNiIjh1Q1zHJsBS21fBJxFdfppoqQ9y/a1Je1Ylg8BNgZeA3y1pmR7REQMk46POKhGEGdJWgEsAz4IPA18pcx3rAV8SdK9VPWwXm/7LklnA18G3rPSzjdPWfWIiFbqeOKwfTVVJdx6r2mwbtua/b7StqAiIqJfHT9VFRERI0vHRxztNufuJfScemXb+l905kFt6zsiohtlxBEREU1J4oiIiKZ0PHFIuryUFplX7vhG0vsk3SFpmqRvlSuokDRR0qWl9MgtkvbubPQREWuebpjjONb2A5LWBW6RdCXwr8BuVM8hvxa4rbT9MvBF2zdK2pLqaqzt6ztMyZGIiPbphsRxkqS3leUtqJ5NfoPtBwAkXcIzl+HuT/Vc8r59N5S0vu1HaztMyZGIiPbpaOIolW73p6o7tVTS9cBvaTCKKEYBr7L9xPBEGBER9To9xzEOeLAkje2AVwHrAa+VtJGktYBDa9pfA5zY90LSpGGNNiIiOp44rgLWkjSfqpzIzcDdwGeBacBNwCKg78HhJwG9kmZLuh04btgjjohYw8nuvimAvnmLMuK4DDjX9mWD6au3t9fTp09vbYAREas5STNs9zba1ukRR39OlzQLmAssBC7vcDwREVF0w1VVz2H7lE7HEBERjXXriKNl5ty9ZNWNIiJiwLoycUgaL+n4ZrdFRET7dWXiAMYDz0kOZbK84baIiBgeXTnHQXVp7tZlgnwZ8ATwILAdcGvNtp/a/kjnwoyIWPN0a+I4FdjJ9qRyd/mV5fVCST192/rbObWqIiLap1tPVdWbZu1AJ/IAAAU6SURBVHvhQBvbnmK713bv6LF55nhERCuNlMTxWKcDiIiISrcmjkeADQaxLSIi2qwr5zhs3y/pJklzgceBe/vZ9pNMjkdEDK+uTBwAto8YzLZ6O2+eOY6IiFbq1lNVERHRpZI4IiKiKUkcERHRlCSOiIhoShJHREQ0JYkjIiKaksQRERFNSeKIiIimJHFERERTZLvTMbSVpEeABZ2OYyUmAIs7HcQqdHuMiW9ouj0+6P4YV8f4Xmy74XMpurbkSAstsN3b6SD6I2l6N8cH3R9j4huabo8Puj/GNS2+nKqKiIimJHFERERT1oTEMaXTAaxCt8cH3R9j4huabo8Puj/GNSq+1X5yPCIiWmtNGHFEREQLJXFERERTRnTikHSApAWSfi/p1Abb15H0/bL9N5J6arZ9vKxfIOkfuik+SZtIuk7So5LObkdsQ4zvDZJmSJpTfu/XZfHtIWlW+blN0tvaEd9QYqzZvmX573xKN8UnqUfS4zWf4ze6Kb6ybRdJv5Y0r/xbHNMt8Uk6suazmyVphaRJrY5viDGuLen88tnNl/TxAR/U9oj8AUYDfwC2Ap4H3AbsUNfmeOAbZfmdwPfL8g6l/TrAS0o/o7sovvWAfYDjgLO78PN7ObBZWd4JuLvL4hsLrFWWNwX+1ve6W2Ks2T4VuAQ4pZviA3qAue34t9ei+NYCZgO7ltebdNP/w3Vtdgb+0IWf4RHA98ryWGAR0DOQ447kEccewO9t32n7KeB7wFvr2rwVOL8sTwVeL0ll/fdsP2l7IfD70l9XxGf7Mds3Ak+0OKZWxTfT9j1l/TxgXUnrdFF8S20/XdaPAdp1BchQ/g0i6WBgIdVn2HXxDYOhxPdGYLbt2wBs3297eRfFV+tdZd92GEqMBtaTtBawLvAU8PBADjqSE8fmwF01r/9c1jVsU/6QLKH6ZjKQfTsZ33BoVXyHArfafrKb4pP0SknzgDnAcTWJpCtilLQ+8DHgjDbENeT4yraXSJop6QZJr+6y+LYFLOlqSbdK+miXxVfrcOC7bYhvqDFOBR4D/gL8Cfh/th8YyEHXhJIj0SaSdgQ+T/Xtr6vY/g2wo6TtgfMl/cR2O0dwzTod+KLtR4fvC35T/gJsaft+Sa8ALpe0o+0BfSMdBmtRnc7dHVgK/FzSDNs/72xYzybplcBS23M7HUsDewDLgc2AjYBfSvqZ7TtXteNIHnHcDWxR8/pFZV3DNmU4Ng64f4D7djK+4TCk+CS9CLgMONr2H7otvj625wOPUs3FdFOMrwS+IGkRcDLwL5I+1C3xldO49wPYnkF1Hn3bbomP6pv1L2wvtr0U+DGwWxfF1+edtG+0MdQYjwCusr3M9t+Am4CB1bNqx4TNcPxQfeO4k2pyu29SaMe6Nifw7EmhH5TlHXn25PidtH5ibdDx1Ww/hvZNjg/l8xtf2h/Spf99X8Izk+MvBu4BJnRTjHVtTqc9k+ND+Qwn9v0/QTXxejewcRfFtxFwK+VCCOBnwEHdEl95Pap8blu1+r9tiz7DjwHfLsvrAbcDuwzouO16Q8PxAxwI3EH1begTZd2ngLeU5TFUV6z8HphW+x8Q+ETZbwHwpi6MbxHwANW35T9Td6VEJ+MDTqM6Nzqr5uf5XRTfu6kmnGeVPy4Hd+O/wZo+TqcNiWOIn+GhdZ/hP3ZTfGXbUSXGucAXujC+fYGb2/VvrwX/jdcv6+dRJY2PDPSYKTkSERFNGclzHBER0QFJHBER0ZQkjoiIaEoSR0RENCWJIyIimpLEERERTUniiIiIpvx/6F2pH0q9/qUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "vimps = np.array(forest.vimp(model).rx('importance')[0])\n",
    "\n",
    "y = np.arange(len(vimps))\n",
    "plt.barh(y, np.abs(vimps))\n",
    "plt.yticks(y, df_train.drop(['time', 'status'], axis=1).columns)\n",
    "plt.title(\"VIMP (absolute value)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2YGhK2xwjkiA"
   },
   "source": [
    "### Question:\n",
    "\n",
    "How does the variable importance compare to that of the Cox model? Which variable is important in both models? Which variable is important in the random survival forest but not in the Cox model? You should see that `edema` is important in both the random survival forest and the Cox model. You should also see that `bili` is important in the random survival forest but not the Cox model ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "\n",
    "You've finished the last assignment in course 2! Take a minute to look back at the analysis you've done over the last four assignments. You've done a great job!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "C2_W4_Assignment.ipynb",
   "provenance": []
  },
  "coursera": {
   "schema_names": [
    "AI4MC2-4"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}