Cleaning Functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains the class 'cleaining' that has several methods that are useful to clean categorical variables. The methods are the following:\n",
    "\n",
    "- <b/>get_nulls(dataframe, columns)</b>: This method returns a dictionary with the percentage of nulls of each columns of a dataframe.\n",
    "  \n",
    "    -Inputs:\n",
    "    \n",
    "        - dataframe: a pandas dataframe object.\n",
    "        - columns: the columns of the dataframe to be included in the calculation. If this is not specified all the \n",
    "          columns will be taken into account.\n",
    "          \n",
    "- <b/>remove_nulls(dataframe, cut_off, columns)</b>: This method remove the columns of a dataframe that have a percentage of nulls higher than a certain cut_off percentage of nulls.\n",
    "\n",
    "    -Inputs:\n",
    "    \n",
    "        - dataframe: a pandas dataframe object.\n",
    "        - cut_off: The minimum percentage of nulls allowed to keep a columns. If a column has a percentage of nulls higher \n",
    "          than the cut_off percentage, it will be removed.\n",
    "        - columns: the columns of the dataframe to be included in the operation. If this is not specified all the \n",
    "          columns will be taken into account.\n",
    "          \n",
    "- <b/>fill_nulls(dataframe, label, columns)</b>: This method fill the null values of the columns of a dataframe with a desired label.\n",
    "\n",
    "    -Inputs:\n",
    "    \n",
    "        - dataframe: a pandas dataframe object.\n",
    "        - label: The text that will be used to replace nulls.\n",
    "        - columns: the columns of the dataframe to be included in the operation. If this is not specified all the \n",
    "          columns will be taken into account.\n",
    "          \n",
    "       \n",
    "- <b/>group_categories(dataframe, cut_off, label, columns)</b>: This method change the category of a categorical variable to a desired label if the percentage of occurence of the category is less than a certain cut_off percentage. This allows to put in the same category those categories with low frequency.\n",
    "\n",
    "    -Inputs:\n",
    "    \n",
    "        - dataframe: a pandas dataframe object.\n",
    "        - cut_off: Categories with a percentage of occurence less than the cut_off percenatage will be relabeled\n",
    "        - label: The label for those categories that will be relabeled.\n",
    "        - columns: the columns of the dataframe to be included in the operation. If this is not specified all the \n",
    "          columns will be taken into account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print all the outputs in a cell\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "class cleaning(object):\n",
    "    \n",
    "    @staticmethod\n",
    "    def get_nulls(dataframe, columns=None):\n",
    "        if columns==None: \n",
    "            columns=dataframe.columns.tolist()\n",
    "        nulls={}\n",
    "        for feat in columns:\n",
    "            nulls[feat]=(dataframe[feat].isnull().sum()*1.0/len(dataframe[feat]))*100\n",
    "    \n",
    "        return nulls\n",
    "    \n",
    "    @staticmethod\n",
    "    def remove_nulls(dataframe, cut_off=5 ,columns=None):\n",
    "        n_dataframe=dataframe.copy()\n",
    "        if columns==None: \n",
    "            columns=dataframe.columns.tolist()\n",
    "        else:\n",
    "            n_dataframe=n_dataframe[columns]\n",
    "        \n",
    "        nulls=cleaning().get_nulls(n_dataframe, columns)\n",
    "        keep_cat = []\n",
    "        for cat_var in nulls:\n",
    "            if nulls[cat_var]<cut_off:\n",
    "                keep_cat.append(cat_var)\n",
    "\n",
    "        return n_dataframe[keep_cat]\n",
    "    \n",
    "    @staticmethod\n",
    "    def fill_nulls(dataframe, label='no data' ,columns=None):\n",
    "        f_dataframe=dataframe.copy()\n",
    "        if columns==None: \n",
    "            columns=dataframe.columns.tolist()\n",
    "        else:\n",
    "            f_dataframe=f_dataframe[columns]\n",
    "    \n",
    "        nulls=cleaning().get_nulls(f_dataframe, columns)\n",
    "\n",
    "        for col in nulls:\n",
    "             if nulls[col]>0:   \n",
    "                f_dataframe[col]=f_dataframe[col].fillna(value=label)\n",
    "\n",
    "        return f_dataframe\n",
    "    \n",
    "    @staticmethod\n",
    "    def group_categories(dataframe, cut_off, label='Other', columns=None):\n",
    "        g_dataframe=dataframe.copy()\n",
    "        if columns==None: \n",
    "            columns=dataframe.columns.tolist()\n",
    "    \n",
    "        g_dataframe = cleaning().fill_nulls(g_dataframe,columns=columns)\n",
    "        \n",
    "        for col in columns:\n",
    "            for category in g_dataframe[col].unique():\n",
    "                if (g_dataframe[col].value_counts()/len(g_dataframe[col])*100)[category]<cut_off:\n",
    "                    g_dataframe[col][g_dataframe[col]==category]=label\n",
    "                \n",
    "        return g_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the examples, we will use a dataset with a lot of categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. get_nulls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will check the percentage of nulls of each column in the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'age': 0.0,\n",
       " 'business_or_self_employed': 0.0,\n",
       " 'capital_gains': 0.0,\n",
       " 'capital_losses': 0.0,\n",
       " 'citizenship': 0.0,\n",
       " 'class_of_worker': 0.0,\n",
       " 'country_father': 3.3645243906717517,\n",
       " 'country_mother': 3.0668143522300686,\n",
       " 'country_self': 1.7005558256441615,\n",
       " 'd_household_family_stat': 0.0,\n",
       " 'd_household_summary': 0.0,\n",
       " 'dividend_from_Stocks': 0.0,\n",
       " 'education': 0.0,\n",
       " 'enrolled_in_edu_inst_lastwk': 0.0,\n",
       " 'family_members_under_18': 0.0,\n",
       " 'fill_questionnaire_veteran_admin': 0.0,\n",
       " 'full_parttime_employment_stat': 0.0,\n",
       " 'hispanic_origin': 0.43804473669702243,\n",
       " 'income_level': 0.0,\n",
       " 'industry_code': 0.0,\n",
       " 'live_1_year_ago': 0.0,\n",
       " 'major_industry_code': 0.0,\n",
       " 'major_occupation_code': 0.0,\n",
       " 'marital_status': 0.0,\n",
       " 'member_of_labor_union': 0.0,\n",
       " 'migration_msa': 49.967171704515266,\n",
       " 'migration_reg': 49.967171704515266,\n",
       " 'migration_sunbelt': 49.967171704515266,\n",
       " 'migration_within_reg': 49.967171704515266,\n",
       " 'num_person_Worked_employer': 0.0,\n",
       " 'occupation_code': 0.0,\n",
       " 'race': 0.0,\n",
       " 'reason_for_unemployment': 0.0,\n",
       " 'region_of_previous_residence': 0.0,\n",
       " 'sex': 0.0,\n",
       " 'state_of_previous_residence': 0.35484630844564286,\n",
       " 'tax_filer_status': 0.0,\n",
       " 'veterans_benefits': 0.0,\n",
       " 'wage_per_hour': 0.0,\n",
       " 'weeks_worked_in_year': 0.0,\n",
       " 'year': 0.0}"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning().get_nulls(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. remove_nulls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we will remove the columns that have more than 10% of null values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new = cleaning().remove_nulls(df, cut_off=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'age': 0.0,\n",
       " 'business_or_self_employed': 0.0,\n",
       " 'capital_gains': 0.0,\n",
       " 'capital_losses': 0.0,\n",
       " 'citizenship': 0.0,\n",
       " 'class_of_worker': 0.0,\n",
       " 'country_father': 3.3645243906717517,\n",
       " 'country_mother': 3.0668143522300686,\n",
       " 'country_self': 1.7005558256441615,\n",
       " 'd_household_family_stat': 0.0,\n",
       " 'd_household_summary': 0.0,\n",
       " 'dividend_from_Stocks': 0.0,\n",
       " 'education': 0.0,\n",
       " 'enrolled_in_edu_inst_lastwk': 0.0,\n",
       " 'family_members_under_18': 0.0,\n",
       " 'fill_questionnaire_veteran_admin': 0.0,\n",
       " 'full_parttime_employment_stat': 0.0,\n",
       " 'hispanic_origin': 0.43804473669702243,\n",
       " 'income_level': 0.0,\n",
       " 'industry_code': 0.0,\n",
       " 'live_1_year_ago': 0.0,\n",
       " 'major_industry_code': 0.0,\n",
       " 'major_occupation_code': 0.0,\n",
       " 'marital_status': 0.0,\n",
       " 'member_of_labor_union': 0.0,\n",
       " 'num_person_Worked_employer': 0.0,\n",
       " 'occupation_code': 0.0,\n",
       " 'race': 0.0,\n",
       " 'reason_for_unemployment': 0.0,\n",
       " 'region_of_previous_residence': 0.0,\n",
       " 'sex': 0.0,\n",
       " 'state_of_previous_residence': 0.35484630844564286,\n",
       " 'tax_filer_status': 0.0,\n",
       " 'veterans_benefits': 0.0,\n",
       " 'wage_per_hour': 0.0,\n",
       " 'weeks_worked_in_year': 0.0,\n",
       " 'year': 0.0}"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning().get_nulls(df_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the method with just some of the columns of the dataframe, in this case the default cut_off value is 5%:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_new2 = cleaning().remove_nulls(df,columns=['region_of_previous_residence','migration_msa','migration_sunbelt',\n",
    "                                             'wage_per_hour'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'region_of_previous_residence': 0.0, 'wage_per_hour': 0.0}"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleaning().get_nulls(df_new2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. fill_nulls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will replace the nulls of each column with a label. First let's check some variables with nulls:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11    NaN\n",
       "26    NaN\n",
       "34    NaN\n",
       "54    NaN\n",
       "87    NaN\n",
       "Name: country_father, dtype: object"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11    NaN\n",
       "26    NaN\n",
       "54    NaN\n",
       "87    NaN\n",
       "92    NaN\n",
       "Name: country_mother, dtype: object"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11     NaN\n",
       "87     NaN\n",
       "92     NaN\n",
       "129    NaN\n",
       "193    NaN\n",
       "Name: country_self, dtype: object"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.country_father[df.country_father.isnull()].head()\n",
    "df.country_mother[df.country_mother.isnull()].head()\n",
    "df.country_self[df.country_self.isnull()].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the method fill_nulls, by default the label to replace nulls is 'no data', and check the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11    NaN\n",
       "26    NaN\n",
       "34    NaN\n",
       "54    NaN\n",
       "87    NaN\n",
       "Name: country_father, dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11    NaN\n",
       "26    NaN\n",
       "54    NaN\n",
       "87    NaN\n",
       "92    NaN\n",
       "Name: country_mother, dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11     NaN\n",
       "87     NaN\n",
       "92     NaN\n",
       "129    NaN\n",
       "193    NaN\n",
       "Name: country_self, dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11    no data\n",
       "26    no data\n",
       "34    no data\n",
       "54    no data\n",
       "87    no data\n",
       "Name: country_father, dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11    no data\n",
       "26    no data\n",
       "54    no data\n",
       "87    no data\n",
       "92    no data\n",
       "Name: country_mother, dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "11     no data\n",
       "87     no data\n",
       "92     no data\n",
       "129    no data\n",
       "193    no data\n",
       "Name: country_self, dtype: object"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_filled = cleaning().fill_nulls(df)\n",
    "\n",
    "df.country_father[df.country_father.isnull()].head()\n",
    "df.country_mother[df.country_mother.isnull()].head()\n",
    "df.country_self[df.country_self.isnull()].head()\n",
    "\n",
    "df_filled.country_father[df_filled.country_father=='no data'].head()\n",
    "df_filled.country_mother[df_filled.country_mother=='no data'].head()\n",
    "df_filled.country_self[df_filled.country_self=='no data'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. group_categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will use the method group_categories to change the category of all those categories in the columns 'country_father','country_mother' and 'country_self' that have a percentage of occurence (frequency) less than 2%:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_grouped=cleaning().group_categories(df,cut_off=2,columns=['country_father','country_mother',\n",
    "                                                  'country_self'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see the original columns and the new columns with the new categories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United-States    159163\n",
       "Mexico            10008\n",
       "Puerto-Rico        2680\n",
       "Italy              2212\n",
       "Canada             1380\n",
       "Name: country_father, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    160479\n",
       "Mexico             9781\n",
       "Puerto-Rico        2473\n",
       "Italy              1844\n",
       "Canada             1451\n",
       "Name: country_mother, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    176989\n",
       "Mexico             5767\n",
       "Puerto-Rico        1400\n",
       "Germany             851\n",
       "Philippines         845\n",
       "Name: country_self, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    159163\n",
       "Other             23639\n",
       "Mexico            10008\n",
       "no data            6713\n",
       "Name: country_father, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    160479\n",
       "Other             23144\n",
       "Mexico             9781\n",
       "no data            6119\n",
       "Name: country_mother, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    176989\n",
       "Other             16767\n",
       "Mexico             5767\n",
       "Name: country_self, dtype: int64"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.country_father.value_counts().head()\n",
    "df.country_mother.value_counts().head()\n",
    "df.country_self.value_counts().head()\n",
    "\n",
    "df_grouped.country_father.value_counts().head()\n",
    "df_grouped.country_mother.value_counts().head()\n",
    "df_grouped.country_self.value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit Pipeline Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will implement some of the methods of the class <b>cleaning</b> as classes that can be used in a Scikit Pipeline to transform data. In order to do that we have to create a fit and transform function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "class RemoveNulls( BaseEstimator, TransformerMixin): \n",
    "    def __init__( self, cut_off=5, columns=None): \n",
    "        self.columns = columns\n",
    "        self.cut_off = cut_off\n",
    "    def get_nulls(self,X):\n",
    "        columns = self.columns\n",
    "        if columns==None: \n",
    "            columns=X.columns.tolist()\n",
    "        nulls={}\n",
    "        for feat in columns:\n",
    "            nulls[feat]=(X[feat].isnull().sum()*1.0/len(X[feat]))*100   \n",
    "        return nulls\n",
    "    def fit( self, X, y = None): \n",
    "        return self \n",
    "    def transform( self, X): \n",
    "        n_dataframe=X.copy()\n",
    "        columns = self.columns\n",
    "        if columns==None: \n",
    "            columns=X.columns.tolist()\n",
    "        else:\n",
    "            n_dataframe=n_dataframe[columns]\n",
    "        \n",
    "        nulls=self.get_nulls(X)\n",
    "        keep_cat = []\n",
    "        for cat_var in nulls:\n",
    "            if nulls[cat_var]<self.cut_off:\n",
    "                keep_cat.append(cat_var)\n",
    "\n",
    "        return n_dataframe[keep_cat]    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GroupCategories( BaseEstimator, TransformerMixin): \n",
    "    def __init__( self, cut_off=5, label='Other', columns=None): \n",
    "        self.columns = columns\n",
    "        self.label=label\n",
    "        self.cut_off = cut_off\n",
    "    def get_nulls(self,X):\n",
    "        columns = self.columns\n",
    "        if columns==None: \n",
    "            columns=X.columns.tolist()\n",
    "        nulls={}\n",
    "        for feat in columns:\n",
    "            nulls[feat]=(X[feat].isnull().sum()*1.0/len(X[feat]))*100   \n",
    "        return nulls\n",
    "    def fill_nulls(self, X, label='no data'):\n",
    "        f_dataframe=X.copy()\n",
    "        columns=self.columns\n",
    "        if columns==None: \n",
    "            columns=X.columns.tolist()\n",
    "        #else:\n",
    "        #    f_dataframe=f_dataframe[columns]\n",
    "        nulls=self.get_nulls(X[columns])\n",
    "        for col in nulls:\n",
    "             if nulls[col]>0:   \n",
    "                f_dataframe[col]=f_dataframe[col].fillna(value=label)\n",
    "        return f_dataframe\n",
    "    def fit( self, X, y = None): \n",
    "        return self \n",
    "    def transform( self, X): \n",
    "        g_dataframe=X.copy()\n",
    "        columns=self.columns\n",
    "        if columns==None: \n",
    "            columns=dataframe.columns.tolist()\n",
    "    \n",
    "        g_dataframe = self.fill_nulls(X)\n",
    "        \n",
    "        for col in columns:\n",
    "            for category in g_dataframe[col].unique():\n",
    "                if (g_dataframe[col].value_counts()/len(g_dataframe[col])*100)[category]<self.cut_off:\n",
    "                    g_dataframe[col][g_dataframe[col]==category]=self.label\n",
    "                \n",
    "        return g_dataframe\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's check the transformations using Scikit's pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "\n",
    "pipeline = Pipeline([('Remove nulls', RemoveNulls(10)),\n",
    "                     ('Group Categories', GroupCategories(cut_off=2,\n",
    "                        columns=['country_father','country_mother', 'country_self']))])\n",
    "\n",
    "df_new = pipeline.fit_transform(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check remaining columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['member_of_labor_union',\n",
       " 'country_mother',\n",
       " 'enrolled_in_edu_inst_lastwk',\n",
       " 'capital_gains',\n",
       " 'business_or_self_employed',\n",
       " 'fill_questionnaire_veteran_admin',\n",
       " 'veterans_benefits',\n",
       " 'dividend_from_Stocks',\n",
       " 'region_of_previous_residence',\n",
       " 'sex',\n",
       " 'major_occupation_code',\n",
       " 'weeks_worked_in_year',\n",
       " 'family_members_under_18',\n",
       " 'class_of_worker',\n",
       " 'd_household_summary',\n",
       " 'capital_losses',\n",
       " 'live_1_year_ago',\n",
       " 'marital_status',\n",
       " 'd_household_family_stat',\n",
       " 'reason_for_unemployment',\n",
       " 'num_person_Worked_employer',\n",
       " 'industry_code',\n",
       " 'education',\n",
       " 'wage_per_hour',\n",
       " 'citizenship',\n",
       " 'income_level',\n",
       " 'hispanic_origin',\n",
       " 'country_self',\n",
       " 'major_industry_code',\n",
       " 'race',\n",
       " 'full_parttime_employment_stat',\n",
       " 'year',\n",
       " 'occupation_code',\n",
       " 'age',\n",
       " 'tax_filer_status',\n",
       " 'state_of_previous_residence',\n",
       " 'country_father']"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new.columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check grouped columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United-States    159163\n",
       "Other             23639\n",
       "Mexico            10008\n",
       "no data            6713\n",
       "Name: country_father, dtype: int64"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    160479\n",
       "Other             23144\n",
       "Mexico             9781\n",
       "no data            6119\n",
       "Name: country_mother, dtype: int64"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "United-States    176989\n",
       "Other             16767\n",
       "Mexico             5767\n",
       "Name: country_self, dtype: int64"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new.country_father.value_counts().head()\n",
    "df_new.country_mother.value_counts().head()\n",
    "df_new.country_self.value_counts().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}