Speed Dating Match Prediction.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SCZaTk6POkXF"
      },
      "source": [
        "# Defining the problem"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EG7iwHNbO_HM"
      },
      "source": [
        "### What is the input?\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LI3RWLu5POXz"
      },
      "source": [
        "the input data representing the outcome of speed dating session based on the profile of two people\n",
        "\n",
        "*   conisting of 191 features that represent each speed dating session \n",
        "*   The dataset is clean, but has a lot of missing values which need to preprocessed\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lh4P8Pf2RtC7"
      },
      "source": [
        "### What is the output?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfogqJlZSBOO"
      },
      "source": [
        "The output is the probability of matching people \n",
        "\n",
        "\n",
        "*   The prediction of the probability (0-1, float) that the dating session will lead to a successful match.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V88cNuLOSp7e"
      },
      "source": [
        "### What data mining function is required?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7P-930fTHrI"
      },
      "source": [
        "As I understand from this part of the slide\n",
        "\n",
        "\n",
        "```\n",
        "Data Mining Functions\n",
        "1. Generalization and Summarization\n",
        "2. Association and Correlation\n",
        "3. Classification & Prediction\n",
        "4. Clustering\n",
        "5. Outlier/Anomaly Analysis\n",
        "6. Time and Ordering \n",
        "7. Structure and Network Analysis\n",
        "```\n",
        "\n",
        "The data mining in this problem requires Classification & Prediction After preprocessing the data\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cNfm22JTIs9"
      },
      "source": [
        "### What could be the challenges?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SgXlNDGyTPEt"
      },
      "source": [
        "The Challenges represented in:\n",
        "\n",
        "\n",
        "*   Missing data\n",
        "*   the dataset is highly unbalanced (mostly unmatched)\n",
        "*   Searching for the best Hyperparameters \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wUrv-fxXTPWj"
      },
      "source": [
        "### What is the impact?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPmpbe4wTYUz"
      },
      "source": [
        "The impact of using the raw data as it is, without cleaning and reprocessing, will result a model with low accuracy that doesn't learn well or as desired from the data in the traing stage\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "the real-life impact of building a model that solve this problem is represent in implementing a recommendation system to better match people in speed dating events according to the result of some survey that people could fill about themselves and their possible partener and according to the previous collected data.\n",
        "<br/>\n",
        "and this will save the time that they may wast together if they aren't matching \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4yjhAPTaTYgP"
      },
      "source": [
        "### What is an ideal solution?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "smtxsp6rTe6q"
      },
      "source": [
        "the ideal solution is to clean and preprocess the data before working with it\n",
        "\n",
        "\n",
        "\n",
        "> Some of the possible solutions are:\n",
        "\n",
        "\n",
        "\n",
        "*   Filling the missing data with approprait value according to the result of hyperparamter tuning \n",
        "*   Drop useless Columns (or Features) in order to reduce the dimintionality (Feature selection)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yzijuPO0fkKC"
      },
      "source": [
        "### What is the experimental protocol used and how was it carried out? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7t0TqNx_fqp3"
      },
      "source": [
        "After loading the data and cleaning and preprocessing it, the experimental protocol used is setting value for k-fold cross validation cv while using GridSearchCV / RandomizedSearchCV / or BayesSearchCV\n",
        "<br/>\n",
        "and measure the perormance using (roc_auc)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Qo-uOPBfrje"
      },
      "source": [
        "### What preprocessing steps are used?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KZW8oroKfysp"
      },
      "source": [
        "*   view the data and understand it\n",
        "*   using df.info() to get more insight about the data\n",
        "*   check the missing data using df.isna().sum()\n",
        "*   convert all object columns to categorical column\n",
        "*   extracting numeric features and categorical features\n",
        "*   define a pipe line for numeric feature preprocessing with applying StandardScaler on it\n",
        "*   define a pipe line for categorical feature preprocessing with applying OneHotEncoder on it\n",
        "*   define the preprocessor and specify what are the categorical and numeric pipeline on it \n",
        "\n",
        "*   using hyperparameter tuning, try to find: \n",
        "  -   the approprait strategy to fill the missing data\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2hzdweU6Bh7i"
      },
      "source": [
        "# Get Started (Importing packages & Loading the data)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-XrzCH4z6_ss"
      },
      "source": [
        "## Import packages "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FU-40aFA048r",
        "outputId": "38dbb367-dc1b-4987-b285-9fc08ed91b03"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting scikit-optimize\n",
            "  Downloading scikit_optimize-0.9.0-py2.py3-none-any.whl (100 kB)\n",
            "\u001b[K     |████████████████████████████████| 100 kB 3.3 MB/s \n",
            "\u001b[?25hCollecting pyaml>=16.9\n",
            "  Downloading pyaml-21.10.1-py2.py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.21.5)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.1.0)\n",
            "Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.4.1)\n",
            "Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize) (1.0.2)\n",
            "Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from pyaml>=16.9->scikit-optimize) (3.13)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->scikit-optimize) (3.1.0)\n",
            "Installing collected packages: pyaml, scikit-optimize\n",
            "Successfully installed pyaml-21.10.1 scikit-optimize-0.9.0\n"
          ]
        }
      ],
      "source": [
        "# this line is for BayesSearchCV and using skopt package\n",
        "!pip install scikit-optimize"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WkKhg1LT5mqX"
      },
      "outputs": [],
      "source": [
        "import numpy as np \n",
        "import pandas as pd \n",
        "import seaborn as sns \n",
        "import matplotlib.pyplot as plt \n",
        "%matplotlib inline\n",
        "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV, KFold, RandomizedSearchCV\n",
        "from sklearn.metrics import classification_report, confusion_matrix, roc_curve, precision_score,recall_score, f1_score,precision_recall_curve\n",
        "sns.set()\n",
        "from sklearn.pipeline import make_pipeline\n",
        "from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder\n",
        "from sklearn.svm import SVC\n",
        "from time import time\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "from sklearn.compose import ColumnTransformer\n",
        "from sklearn.datasets import fetch_openml\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from xgboost.sklearn import XGBClassifier\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "\n",
        "from skopt import BayesSearchCV\n",
        "from skopt.space import Real, Categorical, Integer\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "from sklearn.model_selection import PredefinedSplit"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BW603T0O67P9"
      },
      "outputs": [],
      "source": [
        "pd.set_option(\"display.max_rows\", 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CZOszu8Y7p1k"
      },
      "source": [
        "## Load Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GxAKKxHY7voE"
      },
      "outputs": [],
      "source": [
        "# Loading the data from csv files\n",
        "\n",
        "train = pd.read_csv('data/train.csv')\n",
        "test = pd.read_csv('data/test.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 236
        },
        "id": "bJ6A7dMT7wh0",
        "outputId": "eb248e38-c611-47a0-b819-a4980794e4c0"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-d989f636-593c-466a-a798-cfc7548703c1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>gender</th>\n",
              "      <th>idg</th>\n",
              "      <th>condtn</th>\n",
              "      <th>wave</th>\n",
              "      <th>round</th>\n",
              "      <th>position</th>\n",
              "      <th>positin1</th>\n",
              "      <th>order</th>\n",
              "      <th>partner</th>\n",
              "      <th>pid</th>\n",
              "      <th>...</th>\n",
              "      <th>sinc3_3</th>\n",
              "      <th>intel3_3</th>\n",
              "      <th>fun3_3</th>\n",
              "      <th>amb3_3</th>\n",
              "      <th>attr5_3</th>\n",
              "      <th>sinc5_3</th>\n",
              "      <th>intel5_3</th>\n",
              "      <th>fun5_3</th>\n",
              "      <th>amb5_3</th>\n",
              "      <th>id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>14</td>\n",
              "      <td>18</td>\n",
              "      <td>2</td>\n",
              "      <td>2.0</td>\n",
              "      <td>14</td>\n",
              "      <td>12</td>\n",
              "      <td>372.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2583</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>14</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>10</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>63.0</td>\n",
              "      <td>...</td>\n",
              "      <td>8.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>7.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6830</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>14</td>\n",
              "      <td>1</td>\n",
              "      <td>13</td>\n",
              "      <td>10</td>\n",
              "      <td>8</td>\n",
              "      <td>8.0</td>\n",
              "      <td>10</td>\n",
              "      <td>10</td>\n",
              "      <td>331.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>38</td>\n",
              "      <td>2</td>\n",
              "      <td>9</td>\n",
              "      <td>20</td>\n",
              "      <td>18</td>\n",
              "      <td>13.0</td>\n",
              "      <td>6</td>\n",
              "      <td>7</td>\n",
              "      <td>200.0</td>\n",
              "      <td>...</td>\n",
              "      <td>9.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5508</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>24</td>\n",
              "      <td>2</td>\n",
              "      <td>14</td>\n",
              "      <td>20</td>\n",
              "      <td>6</td>\n",
              "      <td>6.0</td>\n",
              "      <td>20</td>\n",
              "      <td>17</td>\n",
              "      <td>357.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4828</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 192 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d989f636-593c-466a-a798-cfc7548703c1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d989f636-593c-466a-a798-cfc7548703c1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d989f636-593c-466a-a798-cfc7548703c1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "   gender  idg  condtn  wave  round  position  positin1  order  partner  \\\n",
              "0       0    3       2    14     18         2       2.0     14       12   \n",
              "1       1   14       1     3     10         2       NaN      8        8   \n",
              "2       1   14       1    13     10         8       8.0     10       10   \n",
              "3       1   38       2     9     20        18      13.0      6        7   \n",
              "4       1   24       2    14     20         6       6.0     20       17   \n",
              "\n",
              "     pid  ...  sinc3_3  intel3_3  fun3_3  amb3_3  attr5_3  sinc5_3  intel5_3  \\\n",
              "0  372.0  ...      NaN       NaN     NaN     NaN      NaN      NaN       NaN   \n",
              "1   63.0  ...      8.0       8.0     7.0     8.0      NaN      NaN       NaN   \n",
              "2  331.0  ...      NaN       NaN     NaN     NaN      NaN      NaN       NaN   \n",
              "3  200.0  ...      9.0       8.0     8.0     6.0      NaN      NaN       NaN   \n",
              "4  357.0  ...      NaN       NaN     NaN     NaN      NaN      NaN       NaN   \n",
              "\n",
              "   fun5_3  amb5_3    id  \n",
              "0     NaN     NaN  2583  \n",
              "1     NaN     NaN  6830  \n",
              "2     NaN     NaN  4840  \n",
              "3     NaN     NaN  5508  \n",
              "4     NaN     NaN  4828  \n",
              "\n",
              "[5 rows x 192 columns]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Look at first records of the data \n",
        "train.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Z81S8-YdLCr",
        "outputId": "5d1169ee-e13a-49d0-8409-0a7dbfe800d9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5909 entries, 0 to 5908\n",
            "Columns: 192 entries, gender to id\n",
            "dtypes: float64(173), int64(11), object(8)\n",
            "memory usage: 8.7+ MB\n"
          ]
        }
      ],
      "source": [
        "# show the information of the train dataset\n",
        "train.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HIvmueq4_kIk"
      },
      "source": [
        "# Data Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vBTZZBIG8PiX",
        "outputId": "5b2f6e04-396c-41f8-9cc7-60740d6d3f5a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "304971"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# check the weight of missing values\n",
        "train.isna().sum().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vcr8EPwQC8Jq"
      },
      "outputs": [],
      "source": [
        "# Convert all object columns to categorical column (categorical encoding)\n",
        "train[train.select_dtypes(['object']).columns] = train.select_dtypes(['object']).apply(lambda x: x.astype('category'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-orKGstndVUI",
        "outputId": "dc079abc-0baa-4644-a1f9-c63939c95d0b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5909 entries, 0 to 5908\n",
            "Columns: 192 entries, gender to id\n",
            "dtypes: category(8), float64(173), int64(11)\n",
            "memory usage: 8.5 MB\n"
          ]
        }
      ],
      "source": [
        "# show the information of the train dataset\n",
        "train.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OX7Ip5kSWZZg",
        "outputId": "4cd09951-7cad-49d2-f6ea-2a3e63604c58"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "original shape (5909, 190) (5909,)\n"
          ]
        }
      ],
      "source": [
        "# split the train data to features and lable column\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "y = train['match'] \n",
        "X = train.drop(columns=['match', 'id'])\n",
        "print('original shape', X.shape, y.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pIBE-4MoW1e4",
        "outputId": "d75cfc8d-61d2-4d79-b20b-9ed69b0f14fb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "numeric features: ['gender', 'idg', 'condtn', 'wave', 'round', 'position', 'positin1', 'order', 'partner', 'pid', 'int_corr', 'samerace', 'age_o', 'race_o', 'pf_o_att', 'pf_o_sin', 'pf_o_int', 'pf_o_fun', 'pf_o_amb', 'pf_o_sha', 'attr_o', 'sinc_o', 'intel_o', 'fun_o', 'amb_o', 'shar_o', 'like_o', 'prob_o', 'met_o', 'age', 'field_cd', 'race', 'imprace', 'imprelig', 'goal', 'date', 'go_out', 'career_c', 'sports', 'tvsports', 'exercise', 'dining', 'museums', 'art', 'hiking', 'gaming', 'clubbing', 'reading', 'tv', 'theater', 'movies', 'concerts', 'music', 'shopping', 'yoga', 'exphappy', 'expnum', 'attr1_1', 'sinc1_1', 'intel1_1', 'fun1_1', 'amb1_1', 'shar1_1', 'attr4_1', 'sinc4_1', 'intel4_1', 'fun4_1', 'amb4_1', 'shar4_1', 'attr2_1', 'sinc2_1', 'intel2_1', 'fun2_1', 'amb2_1', 'shar2_1', 'attr3_1', 'sinc3_1', 'fun3_1', 'intel3_1', 'amb3_1', 'attr5_1', 'sinc5_1', 'intel5_1', 'fun5_1', 'amb5_1', 'attr', 'sinc', 'intel', 'fun', 'amb', 'shar', 'like', 'prob', 'met', 'match_es', 'attr1_s', 'sinc1_s', 'intel1_s', 'fun1_s', 'amb1_s', 'shar1_s', 'attr3_s', 'sinc3_s', 'intel3_s', 'fun3_s', 'amb3_s', 'satis_2', 'length', 'numdat_2', 'attr7_2', 'sinc7_2', 'intel7_2', 'fun7_2', 'amb7_2', 'shar7_2', 'attr1_2', 'sinc1_2', 'intel1_2', 'fun1_2', 'amb1_2', 'shar1_2', 'attr4_2', 'sinc4_2', 'intel4_2', 'fun4_2', 'amb4_2', 'shar4_2', 'attr2_2', 'sinc2_2', 'intel2_2', 'fun2_2', 'amb2_2', 'shar2_2', 'attr3_2', 'sinc3_2', 'intel3_2', 'fun3_2', 'amb3_2', 'attr5_2', 'sinc5_2', 'intel5_2', 'fun5_2', 'amb5_2', 'you_call', 'them_cal', 'date_3', 'numdat_3', 'num_in_3', 'attr1_3', 'sinc1_3', 'intel1_3', 'fun1_3', 'amb1_3', 'shar1_3', 'attr7_3', 'sinc7_3', 'intel7_3', 'fun7_3', 'amb7_3', 'shar7_3', 'attr4_3', 'sinc4_3', 'intel4_3', 'fun4_3', 'amb4_3', 'shar4_3', 'attr2_3', 'sinc2_3', 'intel2_3', 'fun2_3', 'amb2_3', 'shar2_3', 'attr3_3', 'sinc3_3', 'intel3_3', 'fun3_3', 'amb3_3', 'attr5_3', 'sinc5_3', 'intel5_3', 'fun5_3', 'amb5_3']\n",
            "categorical features: ['field', 'undergra', 'mn_sat', 'tuition', 'from', 'zipcode', 'income', 'career']\n"
          ]
        }
      ],
      "source": [
        "# extracting numeric features and categorical features names\n",
        "\n",
        "# numeric features \n",
        "features_numeric = list(X.select_dtypes(include=['float64', 'int64']))\n",
        "\n",
        "# categorical features \n",
        "features_categorical = list(X.select_dtypes(include=['category']))\n",
        "\n",
        "print('numeric features:', features_numeric)\n",
        "print('categorical features:', features_categorical)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vOkEBzv0WPsX"
      },
      "outputs": [],
      "source": [
        "\n",
        "np.random.seed(0)\n",
        "\n",
        "# define a pipe line for numeric feature preprocessing\n",
        "transformer_numeric = Pipeline(\n",
        "    steps=[\n",
        "        ('imputer', SimpleImputer()),\n",
        "        ('scaler', StandardScaler())]\n",
        ")\n",
        "\n",
        "# define a pipe line for categorical feature preprocessing\n",
        "transformer_categorical = Pipeline(\n",
        "    steps=[\n",
        "        ('imputer', SimpleImputer(strategy='constant')),\n",
        "        ('onehot', OneHotEncoder(handle_unknown='ignore'))\n",
        "    ]\n",
        ")\n",
        "\n",
        "\n",
        "# define the preprocessor \n",
        "# we also specify what are the categorical \n",
        "preprocessor = ColumnTransformer(\n",
        "    transformers=[\n",
        "        ('num', transformer_numeric, features_numeric),\n",
        "        ('cat', transformer_categorical, features_categorical)\n",
        "    ]\n",
        ")\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q9uCc9tKkLdn"
      },
      "source": [
        "# Saving Prediction Result\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WmgD4S2WkTiQ"
      },
      "outputs": [],
      "source": [
        "# define function to save the csv file of the result after each trial\n",
        "def saveResult(test, classifier, fileName):\n",
        "  submission = pd.DataFrame()\n",
        "\n",
        "  submission['id'] = test['id']\n",
        "\n",
        "  submission['match'] = classifier.predict_proba(test.drop(columns=['id']))[:,1]\n",
        "\n",
        "  submission.to_csv(fileName, index=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H02e9eFsdnsY"
      },
      "source": [
        "# Tuning Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hr6flGqxk9g_"
      },
      "source": [
        "## RandomForestClassifier Pipeline with **GridSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-oDQ8b77e-Q"
      },
      "source": [
        "thoughts and observations for trial 0, plan for trial 1: \n",
        "\n",
        "<br/>\n",
        "\n",
        "I used **RandomForestClassifier** through the pipeline that contain the classifier and the preprocessor object which created in the preprocessing step\n",
        "with **GridSearchCV** function to get the best hyperparameters that give the better accuracy.\n",
        "<br/>\n",
        "\n",
        "the hyperparameters that used in this trial:\n",
        "* 'imputer__strategy': ['mean'] => strategey of filling the missing data\n",
        "* 'n_estimators': [20, 30, 40, 50]  \n",
        "* 'max_depth':[5, 10, 20, 30]\n",
        "<br/>\n",
        "\n",
        "I excepected to get the best hyperparameter that reach the global optimal (within the given range) and produce the better accuracy among all combination\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lginwFpi4i17"
      },
      "outputs": [],
      "source": [
        "# combine the preprocessor with the model as a full tunable pipeline\n",
        "full_pipline = Pipeline(\n",
        "    steps=[\n",
        "        ('preprocessor', preprocessor),\n",
        "        ('my_classifier', \n",
        "           RandomForestClassifier(),\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cZ91jdHxW3vu"
      },
      "outputs": [],
      "source": [
        "# fitting the pipeline\n",
        "# The pipeline object can be used like any sk-learn model  \n",
        "full_pipline = full_pipline.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9BNGj5iNW8J_",
        "outputId": "0c1835aa-0f6e-4386-d475-ef71fd5f2b9b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 4 folds for each of 16 candidates, totalling 64 fits\n",
            "best score 0.8436019397678725\n",
            "best score {'my_classifier__max_depth': 5, 'my_classifier__n_estimators': 50, 'preprocessor__num__imputer__strategy': 'mean'}\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# specifying the search space (hyperparameters)\n",
        "param_grid = {\n",
        "    'preprocessor__num__imputer__strategy': ['mean'],\n",
        "    'my_classifier__n_estimators': [20, 30, 40, 50],  \n",
        "    'my_classifier__max_depth':[5, 10, 20, 30]       \n",
        "}\n",
        "\n",
        "# four-fold cross-validation\n",
        "grid_search = GridSearchCV(\n",
        "    full_pipline, param_grid, cv=4, verbose=1, n_jobs=2, \n",
        "    scoring='roc_auc')\n",
        "\n",
        "grid_search.fit(X, y)\n",
        "\n",
        "print('best score {}'.format(grid_search.best_score_))\n",
        "print('best score {}'.format(grid_search.best_params_))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7Tmb0sTkXMTP"
      },
      "outputs": [],
      "source": [
        "# call saveResult function to save the predicted result to csv file\n",
        "saveResult(test, grid_search, 'RandomForestClassifier_Pipeline_with_GridSearchCV.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nZsbEbLnSgCx"
      },
      "source": [
        "## GradientBoostingClassifier Pipeline with **GridSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IibP0l_tu17-"
      },
      "source": [
        "thoughts and observations for trial 1, plan for trial 2: \n",
        "\n",
        "<br/>\n",
        "\n",
        "I used **GradientBoostingClassifier** through the pipeline that contain the classifier and the preprocessor object which created in the preprocessing step\n",
        "with **GridSearchCV** function to get the best hyperparameters that give the better accuracy.\n",
        "<br/>\n",
        "\n",
        "the hyperparameters that used in this trial:\n",
        "* 'imputer__strategy': ['mean', 'median'] => strategey of filling the missing data\n",
        "* 'n_estimators': [250, 500, 750]  \n",
        "* 'max_depth': [3, 5, 7, 9]\n",
        "* 'learning_rate': [0.01, 0.1, 1]  \n",
        "\n",
        "<br/>\n",
        "\n",
        "I excepected to get the best hyperparameter that reach the global optimal (within the given range) and produce the better accuracy among all combination\n",
        "<br/>\n",
        "Also I excepected a better accuracy than the previous trial\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xIUIc08lSgCy"
      },
      "outputs": [],
      "source": [
        "# combine the preprocessor with the model as a full tunable pipeline\n",
        "full_pipline = Pipeline(\n",
        "    steps=[\n",
        "        ('preprocessor', preprocessor),\n",
        "        ('my_classifier', \n",
        "           GradientBoostingClassifier(),\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tCRQ4HDvSgC0"
      },
      "outputs": [],
      "source": [
        "# fitting the pipeline\n",
        "# The pipeline object can be used like any sk-learn model  \n",
        "full_pipline = full_pipline.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KeSgpqKXSgC1",
        "outputId": "050cb714-e531-4064-f54d-5710821e9154"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 3 folds for each of 36 candidates, totalling 108 fits\n",
            "best score 0.8736446887339357\n",
            "best score {'my_classifier__learning_rate': 0.01, 'my_classifier__max_depth': 5, 'my_classifier__n_estimators': 750, 'preprocessor__num__imputer__strategy': 'mean'}\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# specifying the search space (hyperparameters)\n",
        "param_grid = {\n",
        "    'preprocessor__num__imputer__strategy': ['mean', 'median'],\n",
        "    'my_classifier__n_estimators': [250, 500, 750],  \n",
        "    'my_classifier__max_depth': [3, 5, 7, 9],\n",
        "    'my_classifier__learning_rate': [0.01, 0.1, 1]       \n",
        "}\n",
        "\n",
        "# three-fold cross-validation\n",
        "grid_search = GridSearchCV(\n",
        "    full_pipline, param_grid, cv=3, verbose=1, n_jobs=2, \n",
        "    scoring='roc_auc')\n",
        "\n",
        "grid_search.fit(X, y)\n",
        "\n",
        "print('best score {}'.format(grid_search.best_score_))\n",
        "print('best score {}'.format(grid_search.best_params_))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ay6xeNMjSgC3"
      },
      "outputs": [],
      "source": [
        "# call saveResult function to save the predicted result to csv file\n",
        "saveResult(test, grid_search, 'GradientBoostingClassifier_Pipeline_with_GridSearchCV.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "re85W8jMln_A"
      },
      "source": [
        "## XGBClassifier Pipeline with **GridSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8A4GG8LpxXkr"
      },
      "source": [
        "thoughts and observations for trial 2, plan for trial 3: \n",
        "\n",
        "<br/>\n",
        "\n",
        "I used **XGBClassifier** through the pipeline that contain the classifier and the preprocessor object which created in the preprocessing step\n",
        "with **GridSearchCV** function to get the best hyperparameters that give the better accuracy.\n",
        "<br/>\n",
        "\n",
        "the hyperparameters that used in this trial:\n",
        "* 'imputer__strategy': ['mean', 'median', 'most_frequent']  => strategey of filling the missing data\n",
        "* 'n_estimators': [50, 100, 200]  \n",
        "* 'max_depth': [2, 7, 10]  \n",
        "\n",
        "<br/>\n",
        "\n",
        "I excepected to get the best hyperparameter that reach the global optimal (within the given range) and produce the better accuracy among all combination\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mVYZRPC6lqHa"
      },
      "outputs": [],
      "source": [
        "# combine the preprocessor with the model as a full tunable pipeline\n",
        "full_pipline = Pipeline(\n",
        "    steps=[\n",
        "        ('preprocessor', preprocessor),\n",
        "        ('my_classifier', \n",
        "           XGBClassifier(),\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dqJaoFhTlqHd"
      },
      "outputs": [],
      "source": [
        "# fitting the pipeline\n",
        "# The pipeline object can be used like any sk-learn model  \n",
        "full_pipline = full_pipline.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GhLuhf5ZlqHe",
        "outputId": "13e0e985-3f39-43f3-c8ca-c301f632b86d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 4 folds for each of 27 candidates, totalling 108 fits\n",
            "best score 0.881179071152443\n",
            "best score {'my_classifier__max_depth': 7, 'my_classifier__n_estimators': 100, 'preprocessor__num__imputer__strategy': 'mean'}\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# specifying the search space (hyperparameters)\n",
        "param_grid = {\n",
        "    'preprocessor__num__imputer__strategy': ['mean', 'median', 'most_frequent'],\n",
        "    'my_classifier__n_estimators': [50, 100, 200],  \n",
        "    'my_classifier__max_depth':[2, 7, 10]       \n",
        "}\n",
        "\n",
        "# four-fold cross-validation\n",
        "grid_search = GridSearchCV(\n",
        "    full_pipline, param_grid, cv=4, verbose=1, n_jobs=2, \n",
        "    scoring='roc_auc')\n",
        "\n",
        "grid_search.fit(X, y)\n",
        "\n",
        "print('best score {}'.format(grid_search.best_score_))\n",
        "print('best score {}'.format(grid_search.best_params_))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jx9fWnxwmQx3"
      },
      "outputs": [],
      "source": [
        "# call saveResult function to save the predicted result to csv file\n",
        "saveResult(test, grid_search, 'XGBClassifier_Pipeline_with_GridSearchCV.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GRa2fALD_Oyz"
      },
      "source": [
        "## XGBClassifier Pipeline with **RandomizedSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "muyKGBKYx8dV"
      },
      "source": [
        "thoughts and observations for trial 3, plan for trial 4: \n",
        "\n",
        "<br/>\n",
        "\n",
        "I used **XGBClassifier** through the pipeline that contain the classifier and the preprocessor object which created in the preprocessing step\n",
        "with **RandomizedSearchCV** function to get the best random hyperparameters of all the hyperparameter combinations among the specified number of iteration \n",
        "<br/>\n",
        "\n",
        "the hyperparameters that used in this trial:\n",
        "* 'imputer__strategy': ['mean', 'median', 'most_frequent']  => strategey of filling the missing data\n",
        "* 'n_estimators': [50, 100, 200]  \n",
        "* 'max_depth': [2, 7, 10]  \n",
        "\n",
        "<br/>\n",
        "\n",
        "I excepected to get lower accuracy compared to the GridSearchCV with the same classifier\n",
        "<br/>\n",
        "Also I excepected the hyperparameters that reach the *local optimal* (within the given range) and produce the better accuracy among the random selected combinations\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KznGGlft_4Z2"
      },
      "outputs": [],
      "source": [
        "# combine the preprocessor with the model as a full tunable pipeline\n",
        "full_pipline = Pipeline(\n",
        "    steps=[\n",
        "        ('preprocessor', preprocessor),\n",
        "        ('my_classifier', \n",
        "           XGBClassifier(),\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EPJY12pI_4Z3"
      },
      "outputs": [],
      "source": [
        "# fitting the pipeline\n",
        "# The pipeline object can be used like any sk-learn model  \n",
        "full_pipline = full_pipline.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0prh_tyD_gU8",
        "outputId": "2822353f-cfa7-4f55-af69-617765d54b5b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
            "best score 0.8819554153240737\n",
            "best score {'preprocessor__num__imputer__strategy': 'most_frequent', 'my_classifier__n_estimators': 200, 'my_classifier__max_depth': 7}\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# specifying the search space (hyperparameters)\n",
        "param_grid = {\n",
        "    'preprocessor__num__imputer__strategy': ['mean', 'median', 'most_frequent'],\n",
        "    'my_classifier__n_estimators': [50, 100, 200],  \n",
        "    'my_classifier__max_depth':[2, 7, 10]       \n",
        "}\n",
        "\n",
        "# five-fold cross-validation\n",
        "grid_search = RandomizedSearchCV(\n",
        "    full_pipline, param_grid, cv=5, verbose=1, n_jobs=2, \n",
        "    # number of random trials\n",
        "    n_iter=20,\n",
        "    scoring='roc_auc')\n",
        "\n",
        "grid_search.fit(X, y)\n",
        "\n",
        "print('best score {}'.format(grid_search.best_score_))\n",
        "print('best score {}'.format(grid_search.best_params_))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iJYZd9Co_78x"
      },
      "outputs": [],
      "source": [
        "# call saveResult function to save the predicted result to csv file\n",
        "saveResult(test, grid_search, 'XGBClassifier_Pipeline_with_RandomizedSearchCV.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OX9rvlznAFT6"
      },
      "source": [
        "## SVM Pipeline with **RandomizedSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lQbsVnht09o2"
      },
      "source": [
        "thoughts and observations for trial 4, plan for trial 5: \n",
        "\n",
        "<br/>\n",
        "\n",
        "I used **SVM Classifier** through the pipeline that contain the classifier and the preprocessor object which created in the preprocessing step\n",
        "with **RandomizedSearchCV** function to get the best random hyperparameters of all the hyperparameter combinations among the specified number of iteration \n",
        "<br/>\n",
        "\n",
        "the hyperparameters that used in this trial:\n",
        "* 'imputer__strategy': ['mean', 'median', 'most_frequent']  => strategey of filling the missing data\n",
        "* 'kernel': ['linear', 'rbf', 'poly'],\n",
        "* 'C': [0.001, 0.01, 0.1, 1, 10, 100, 200],\n",
        "* 'gamma': [0.1, 0.5, 0.7, 1],\n",
        "* 'degree': [1, 2, 3, 4, 5]  \n",
        "\n",
        "<br/>\n",
        "\n",
        "\n",
        "I excepected the hyperparameters that reach the *local optimal* (within the given range) and produce the better accuracy among the random selected combinations\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "43vP22cpAFT6"
      },
      "outputs": [],
      "source": [
        "# combine the preprocessor with the model as a full tunable pipeline\n",
        "full_pipline = Pipeline(\n",
        "    steps=[\n",
        "        ('preprocessor', preprocessor),\n",
        "        ('my_classifier', \n",
        "           SVC(probability=True, class_weight='balanced'),\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "11Em-qsdAFT6"
      },
      "outputs": [],
      "source": [
        "# fitting the pipeline\n",
        "# The pipeline object can be used like any sk-learn model  \n",
        "full_pipline = full_pipline.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "10dEbqMdAFT6",
        "outputId": "a87035c4-85d4-43db-9061-5e9e93a1eec4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
            "best score 0.8636563477471924\n",
            "best score {'preprocessor__num__imputer__strategy': 'mean', 'my_classifier__kernel': 'poly', 'my_classifier__gamma': 0.1, 'my_classifier__degree': 1, 'my_classifier__C': 0.01}\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# specifying the search space (hyperparameters)\n",
        "param = {\n",
        "    'preprocessor__num__imputer__strategy': ['mean', 'median', 'most_frequent'],\n",
        "    'my_classifier__kernel': ['linear', 'rbf', 'poly'],\n",
        "    'my_classifier__C': [0.001, 0.01, 0.1, 1, 10, 100, 200],\n",
        "    'my_classifier__gamma': [0.1, 0.5, 0.7, 1],\n",
        "    'my_classifier__degree': [1, 2, 3, 4, 5]       \n",
        "}\n",
        "\n",
        "# five-fold cross-validation\n",
        "grid_search = RandomizedSearchCV(\n",
        "    full_pipline, param, cv=5, verbose=1, n_jobs=2, \n",
        "    # number of random trials\n",
        "    n_iter=20,\n",
        "    scoring='roc_auc')\n",
        "\n",
        "grid_search.fit(X, y)\n",
        "\n",
        "print('best score {}'.format(grid_search.best_score_))\n",
        "print('best score {}'.format(grid_search.best_params_))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DwPW0hQNAFT7"
      },
      "outputs": [],
      "source": [
        "# call saveResult function to save the predicted result to csv file\n",
        "saveResult(test, grid_search, 'SVC_Pipeline_with_RandomizedSearchCV.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hq3o1djrgPlc"
      },
      "source": [
        "## SVM Pipeline with **BayesSearchCV**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K2n2AiGk2HnX"
      },
      "source": [
        "thoughts and observations for trial 5, plan for trial 6: \n",
        "\n",
        "<br/>\n",
        "\n",
        "I used **SVM Classifier** through the pipeline that contain the classifier and the preprocessor object which created in the preprocessing step\n",
        "with **BayesSearchCV** function to get the best hyperparameters among the specified number of iteration based on using bayesian learning to predict what is the next hyperparamter values we should try given the current trials\n",
        "<br/>\n",
        "\n",
        "the hyperparameters that used in this trial:\n",
        "* 'imputer__strategy': ['mean', 'median', 'most_frequent']  => strategey of filling the missing data\n",
        "* 'kernel': ['linear', 'rbf', 'poly'],\n",
        "* 'C': [0.001, 0.01, 0.1, 1, 10, 100, 200],\n",
        "* 'gamma': [0.1, 0.5, 0.7, 1],\n",
        "* 'degree': [1, 2, 3, 4, 5]  \n",
        "\n",
        "<br/>\n",
        "\n",
        "I excepected to get better accuracy compared to the RandomizedSearchCV with the same classifier\n",
        "<br/>\n",
        "Also I excepected the hyperparameters that reach the *local optimal* (within the given range) and produce the better accuracy among the generated combinations\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "49xYdzA5gPlc"
      },
      "outputs": [],
      "source": [
        "# combine the preprocessor with the model as a full tunable pipeline\n",
        "full_pipline = Pipeline(\n",
        "    steps=[\n",
        "        ('preprocessor', preprocessor),\n",
        "        ('my_classifier', \n",
        "           SVC(probability=True, class_weight='balanced'),\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qmv_ExxUgPlc"
      },
      "outputs": [],
      "source": [
        "# fitting the pipeline\n",
        "# The pipeline object can be used like any sk-learn model  \n",
        "full_pipline = full_pipline.fit(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NvmWDqh0gPld",
        "outputId": "2d488c6b-0662-47c8-b85f-c8db3a9c6a53"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "Fitting 3 folds for each of 1 candidates, totalling 3 fits\n",
            "best score 0.8582010652056825\n",
            "best score OrderedDict([('my_classifier__C', 0.001), ('my_classifier__degree', 1), ('my_classifier__gamma', 0.7), ('my_classifier__kernel', 'linear'), ('preprocessor__num__imputer__strategy', 'median')])\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# specifying the search space (hyperparameters)\n",
        "param = {\n",
        "    'preprocessor__num__imputer__strategy': ['mean', 'median', 'most_frequent'],\n",
        "    'my_classifier__kernel': ['linear', 'rbf', 'poly'],\n",
        "    'my_classifier__C': [0.001, 0.01, 0.1, 1, 10, 100, 200],\n",
        "    'my_classifier__gamma': [0.1, 0.5, 0.7, 1],\n",
        "    'my_classifier__degree': [1, 2, 3, 4, 5]       \n",
        "}\n",
        "\n",
        "# three-fold cross-validation\n",
        "bayes_search = BayesSearchCV(\n",
        "    full_pipline, param, cv=3, n_iter=30, random_state=0, verbose=1, n_jobs=2, \n",
        "    scoring='roc_auc')\n",
        "\n",
        "bayes_search.fit(X, y)\n",
        "\n",
        "print('best score {}'.format(bayes_search.best_score_))\n",
        "print('best score {}'.format(bayes_search.best_params_))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sC6jzb7vgPle"
      },
      "outputs": [],
      "source": [
        "# call saveResult function to save the predicted result to csv file\n",
        "saveResult(test, bayes_search, 'SVC_Pipeline_with_BayesSearchCV.csv')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "os5YUiTX2UT5"
      },
      "source": [
        "# Questions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_AJ5LIz2Vz4"
      },
      "source": [
        "## Why a simple linear regression model (without any activation function) is not good for classification task, compared to Perceptron/Logistic regression?\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_SKwq3M22WBI"
      },
      "source": [
        "Becuase when using simple linear regression model:\n",
        "*  the predicted value is continuous, not probabilistic\n",
        "*  sensitive to imbalance data when using linear regression for classification\n",
        "\n",
        "*  Linear regression produces a linear hypothesis function. However, in classification problems, our data do not show up in a linear distribution but in a grouped distribution.\n",
        "<br/>\n",
        "\n",
        "<br/>\n",
        "\n",
        "resources: \n",
        "* https://jinglescode.github.io/2019/05/07/why-linear-regression-is-not-suitable-for-classification/\n",
        "* https://ai.plainenglish.io/why-dont-we-approach-to-classification-problems-using-linear-regression-in-machine-learning-8edcca89448\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSVJ0pbN2WNa"
      },
      "source": [
        "## What's a decision tree and how it is different to a logistic regression model?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QALa0PDz2qHU"
      },
      "source": [
        "*  **Decision Tree :**  the most powerful and popular tool for classification and prediction. A Decision tree is a flowchart like tree structure, where each internal node denotes a test on an attribute, each branch represents an outcome of the test, and each leaf node (terminal node) holds a class label. \n",
        "\n",
        "*  the different between decesion tree and logistic regression is: Decision Trees bisect the space into smaller and smaller regions, whereas Logistic Regression fits a single line to divide the space exactly into two\n",
        "\n",
        "The resource of the first part of the question is:\n",
        " https://www.geeksforgeeks.org/decision-tree/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NYEqPag02qKZ"
      },
      "source": [
        "## What's the difference between grid search and random search?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQZEq3Xc20z4"
      },
      "source": [
        "\n",
        "\n",
        "> Grid search\n",
        "* Try out every combination of the parameters:\n",
        "* Computationally expensive\n",
        "* Global optimal (within the given range)\n",
        "* Sklearn: model_selection.GridSearchCV\n",
        "\n",
        "\n",
        "> Random search\n",
        "* Try out a random subset\n",
        "* `good enough`\n",
        "* Local optimal (within the given range)\n",
        "* Efficient (less trials)\n",
        "* Sklearn: model_selection.RandomizedSearchCV\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "<br/>\n",
        "\n",
        "**Random Search** replaces the exhaustive enumeration of all combinations that is used in the **grid Search** by selecting them randomly.\n",
        "\n",
        "<br/>\n",
        "\n",
        "**Grid Search** use all the hyperparameter combinations but **Random Search** limit the number of hyperparameter combinations that are tested\n",
        "\n",
        "<br/>\n",
        "\n",
        "Grid search (global optimal) is expensive when you specify a large search space. Alternatively, random search CV gives local optimal (may be good enough and even more generalizable)\n",
        "\n",
        "<br/>\n",
        "\n",
        "the random search is more faster than the grid search\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e98ZUbZm22OE"
      },
      "source": [
        "## What's the difference between bayesian search and random search?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fanM0fTb22Wk"
      },
      "source": [
        "\n",
        "> Random search\n",
        "* Try out a random subset\n",
        "* `good enough`\n",
        "* Local optimal (within the given range)\n",
        "* Efficient (less trials)\n",
        "* Sklearn: model_selection.RandomizedSearchCV\n",
        "\n",
        "\n",
        "> Bayesian Optimization\n",
        "* As an optimization problem\n",
        "* Trial -> estimated error -> Bayesian model estimates the next\n",
        "parameter to try -> trial -> repeat..\n",
        "* pip install bayesian-optimization\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "<br/>\n",
        "\n",
        "The **Random Search** use random sample of the combinations of hyperparameters to train the model and get the score\n",
        "\n",
        "<br/>\n",
        "\n",
        "The main difference between **Bayesian search** and the other methods is that the tuning algorithm optimizes its parameter selection in each round according to the previous round score. Thus, instead of randomly choosing the next set of parameters, the algorithm optimizes the choice, and likely reaches the best parameter set faster than the previous two methods. Meaning, this method chooses only the relevant search space and discards the ranges that will most likely not deliver the best solution. Thus, it can be beneficial when you have a large amount of data, the learning is slow, and you want to minimize the tuning time.\n",
        "\n",
        "\n",
        "<br/>\n",
        "\n",
        "resource for bayesian search:\n",
        "https://towardsdatascience.com/bayesian-optimization-for-hyperparameter-tuning-how-and-why-655b0ee0b399"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "SCZaTk6POkXF",
        "EG7iwHNbO_HM",
        "Lh4P8Pf2RtC7",
        "V88cNuLOSp7e",
        "3cNfm22JTIs9",
        "wUrv-fxXTPWj",
        "4yjhAPTaTYgP",
        "yzijuPO0fkKC",
        "_Qo-uOPBfrje",
        "2hzdweU6Bh7i",
        "-XrzCH4z6_ss",
        "CZOszu8Y7p1k",
        "HIvmueq4_kIk",
        "Q9uCc9tKkLdn",
        "hr6flGqxk9g_",
        "nZsbEbLnSgCx",
        "re85W8jMln_A",
        "GRa2fALD_Oyz",
        "OX9rvlznAFT6",
        "hq3o1djrgPlc",
        "os5YUiTX2UT5",
        "BSVJ0pbN2WNa"
      ],
      "name": "Assignment 2.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}