03_covid_tweet_informative.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification - Detect & Clean incorrect labels using cleanlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "- Introduction\n",
    "- Problem Description\n",
    "- Background & related work\n",
    "- Data Description\n",
    "- Code\n",
    "- Lessons Learnt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction \n",
    "\n",
    "WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets. The goal of the task is to automatically identify whether an English Tweet related to the novel coronavirus (COVID-19) is informative or not. Such informative Tweets provide information about recovered, suspected, confirmed and death cases as well as location or travel history of the cases. Learn more from [here](http://noisy-text.github.io/2020/covid19tweet-task.html) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/gdrive\n"
     ]
    }
   ],
   "source": [
    "# make your Google Drive accessible\n",
    "from google.colab import drive\n",
    "drive.mount('/content/gdrive', force_remount=True)\n",
    "\n",
    "root_dir = \"/content/gdrive/My Drive\"\n",
    "drive_data = f\"{root_dir}/Colab Notebooks/covid\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 194kB 19.5MB/s \n",
      "\u001b[K     |████████████████████████████████| 12.8MB 199kB/s \n",
      "\u001b[K     |████████████████████████████████| 776.8MB 23kB/s \n",
      "\u001b[K     |████████████████████████████████| 61kB 8.3MB/s \n",
      "\u001b[31mERROR: torchtext 0.9.1 has requirement torch==1.8.1, but you'll have torch 1.7.1 which is incompatible.\u001b[0m\n",
      "\u001b[K     |████████████████████████████████| 51kB 6.9MB/s \n",
      "\u001b[K     |████████████████████████████████| 51kB 7.2MB/s \n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "%pip install fastai -Uq\n",
    "%pip install nbdev -Uq\n",
    "%pip install cleanlab -Uq\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from pathlib import Path\n",
    "from fastai.text.all import *\n",
    "from fastcore.test import * \n",
    "#import fastai; fastai.utils.show_install()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replace_space(text, sep='_'):\n",
    "  return sep.join(text.split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def modify_metadata(column_value, tag_name):\n",
    "  data = f\"{tag_name}_{replace_space(column_value)}\"\n",
    "  data.replace(',', '')\n",
    "  return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq('aut_name_Jeremy_Howard', modify_metadata('Jeremy Howard', 'aut_name'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_text(word):\n",
    "\n",
    "  #Remove urls\n",
    "  urls = re.findall(r'https?://\\S+', word)\n",
    "  if len(urls) > 0:\n",
    "    word = \"\"\n",
    "    return word\n",
    "  \n",
    "  return word.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Config.DEFAULT_CONFIG = {\n",
    "        'data_path': f'{drive_data}',\n",
    "        'data_archive_path': f'{drive_data}',\n",
    "        'model_path': f'{drive_data}' + '/models'\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data_archive_path': '/content/gdrive/My Drive/Colab Notebooks/covid',\n",
       " 'data_path': '/content/gdrive/My Drive/Colab Notebooks/covid',\n",
       " 'model_path': '/content/gdrive/My Drive/Colab Notebooks/covid/models'}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Config.DEFAULT_CONFIG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "covid_tweet = f\"{drive_data}\"; \n",
    "covid = Path(covid_tweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Path.BASE_PATH = covid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#5) [Path('train.tsv'),Path('valid.tsv'),Path('unlabeled_test_with_noise.tsv'),Path('test.tsv'),Path('README.md')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "covid.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tweet-preprocessor\n",
      "  Downloading https://files.pythonhosted.org/packages/17/9d/71bd016a9edcef8860c607e531f30bd09b13103c7951ae73dd2bf174163c/tweet_preprocessor-0.6.0-py3-none-any.whl\n",
      "Installing collected packages: tweet-preprocessor\n",
      "Successfully installed tweet-preprocessor-0.6.0\n"
     ]
    }
   ],
   "source": [
    "%pip install tweet-preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import re\n",
    "def extract_hashtag(text):\n",
    "  hash_tags = re.findall(r'#(\\S+)', text)\n",
    "  if hash_tags: \n",
    "    return ' '.join(hash_tags)\n",
    "  else:\n",
    "    return 'NA'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import preprocessor as prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq(['COVID-19'], extract_hashtag(\"welcome to the new world #COVID-19\"))\n",
    "test_eq(['COVID'], extract_hashtag(\"welcome to the new world #COVID\"))\n",
    "test_eq(['breakingnews', 'developing', 'covid19'], \n",
    "        extract_hashtag(\"welcome to the new world #breakingnews #developing #covid19\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load all training, dev, test and private test set.\n",
    "train_df = pd.read_csv(covid/'train.tsv', delimiter='\\t')\n",
    "valid_df = pd.read_csv(covid/'valid.tsv', names=['Id', 'Text', 'Label'], delimiter='\\t')\n",
    "test_df = pd.read_csv(covid/'test.tsv', names=['Id', 'Text', 'Label'], delimiter='\\t')\n",
    "private_test_df = pd.read_csv(covid/'unlabeled_test_with_noise.tsv', names=['Id', 'Text'], delimiter='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_df = pd.concat([train_df, valid_df, test_df, private_test_df])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6936, 1000, 2000, 12000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_df), len(valid_df), len(test_df), len(private_test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words = ['HTTPURL', '@USER']\n",
    "all_df['Text'] =  all_df['Text'].apply(lambda x: ' '.join([each for each in x.split(' ') if each not in stop_words]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['Text'] = test_df['Text'].apply(lambda x: ' '.join([each for each in x.split(' ') if each not in stop_words]))\n",
    "test_df['HashTag'] = test_df.Text.apply(extract_hashtag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stop_add_hashtag(df):\n",
    "  df['Text'] = df['Text'].apply(lambda x: ' '.join([each for each in x.split(' ') if each not in stop_words]))\n",
    "  df['HashTag'] = df.Text.apply(extract_hashtag)\n",
    "  return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['is_valid'] = False\n",
    "valid_df['is_valid'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_val = pd.concat([train_df, valid_df])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_df['HashTag'] = all_df.Text.apply(extract_hashtag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_val['HashTag'] = train_val.Text.apply(extract_hashtag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Text</th>\n",
       "      <th>Label</th>\n",
       "      <th>is_valid</th>\n",
       "      <th>HashTag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1241490299215634434</td>\n",
       "      <td>Official death toll from #covid19 in the United Kingdom is now GREATER than: Germany + Poland + Switzerland + Austria + Portugal + Greece + Sweden + Finland + Norway + Ireland... COMBINED. UK: 67.5 Million (233 dead) Above group: 185 Million (230 dead) HTTPURL</td>\n",
       "      <td>INFORMATIVE</td>\n",
       "      <td>False</td>\n",
       "      <td>covid19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1245916400981381130</td>\n",
       "      <td>Dearest Mr. President @USER 1,169 coronavirus deaths in the US in 24 hours (?) Covid19 pandemic is an international crime from China - not a nature disasster! Please use your authorities to protect your people and world against China! #ChinaHasToCompensateAll</td>\n",
       "      <td>INFORMATIVE</td>\n",
       "      <td>False</td>\n",
       "      <td>ChinaHasToCompensateAll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1241132432402849793</td>\n",
       "      <td>Latest Updates March 20 ⚠️5274 new cases and 38 new deaths in the United States Illinois: Governo Pritzker issues \"stay at home\" order for all residents New York: Governor Cuomo orders 100% of all non-essential workers to stay home Penns...Source ( /coronavirus/country/us/ )</td>\n",
       "      <td>INFORMATIVE</td>\n",
       "      <td>False</td>\n",
       "      <td>NA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1236107253666607104</td>\n",
       "      <td>真把公主不当干部 BREAKING: 21 people on Grand Princess cruise ship docked off the California coast tested positive for coronavirus, including 19 crew members and two passengers, Vice Pres. Mike Pence says. 24 people tested negative. HTTPURL HTTPURL</td>\n",
       "      <td>INFORMATIVE</td>\n",
       "      <td>False</td>\n",
       "      <td>NA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1239673817552879619</td>\n",
       "      <td>OKLAHOMA CITY — The State Department of Education announced Monday the closure of all K-12 public schools statewide until at least April 6 as the number of COVID-19 cases climb and the risk of community spread grows. HTTPURL</td>\n",
       "      <td>UNINFORMATIVE</td>\n",
       "      <td>False</td>\n",
       "      <td>NA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Id  ...                  HashTag\n",
       "0  1241490299215634434  ...                  covid19\n",
       "1  1245916400981381130  ...  ChinaHasToCompensateAll\n",
       "2  1241132432402849793  ...                       NA\n",
       "3  1236107253666607104  ...                       NA\n",
       "4  1239673817552879619  ...                       NA\n",
       "\n",
       "[5 rows x 5 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_val.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6936"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  return array(a, dtype, copy=False, order=order)\n"
     ]
    }
   ],
   "source": [
    "dls_lm = TextDataLoaders.from_df(all_df, seed=42, text_col=['HashTag', 'Text'], is_lm=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang_learn = language_model_learner(dls_lm, AWD_LSTM, metrics=Perplexity())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.06309573650360108, lr_steep=0.04786301031708717)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9Z3/8dcn+0IWlgRIWAKyqVRA4l4tam1rW6tjta11pSp1amk7nenU/mamnZnOTNvpdNGqUKpirVsramsX7TJ1G63UILghICCQsAay79vn98e9QIgJJJBz70nu+/l45EHuOeee8ybi/eT7/Z7z/Zq7IyIiiSsp3gFERCS+VAhERBKcCoGISIJTIRARSXAqBCIiCU6FQEQkwaXEO8BAjRkzxktKSuIdQ0RkSFm1atVedy/obd+QKwQlJSWUlZXFO4aIyJBiZlv72qeuIRGRBKdCICKS4FQIREQSnAqBiEiCC6wQmNlMM1vT7avOzL7Ux7GnmFmHmV0WVB4REeldYHcNuft6YC6AmSUD24HHex4X3fcd4A9BZRERkb7FqmvofGCTu/d2+9Ji4FFgT4yyiIgMOX9cu5uNe+oDOXesCsGngId6bjSzYuBvgCUxyiEiMuS4O597YBWPvrI9kPMHXgjMLA34GPBIL7t/CHzV3buOcI5FZlZmZmWVlZVBxBQRCa2W9i7aO53cjNRAzh+LJ4svBF5x99297CsFHjYzgDHAh82sw91/2f0gd18GLAMoLS3VkmoiklBqm9sByMscuoXgCnrpFgJw9yn7vzeze4Hf9CwCIiKJrq4lUghyM4P5yA60a8jMsoELgMe6bbvJzG4K8roiIsPJkG4RuHsjMLrHtqV9HHtdkFlERIaqumghCGqMQE8Wi4iE3MGuIRUCEZGEVNsUbNeQCoGISMjVtXQAkJMxBAeLRUTk2NU2t5OdlkxqcjAf2SoEIiIhV9fcHtj4AKgQiIiEXm1ze2DjA6BCICISenUt7YHdOgoqBCIioVfb3KGuIRGRRBYZIwju+V8VAhGRkKtrVteQiEjC6uxy6ls7NFgsIpKoGqIPk2mMQEQkQQU98yioEIiIhNqBCecCml4CVAhEREJNLQIRkQR3YC0CFQIRkcQ0pFsEZjbTzNZ0+6ozsy/1OOZKM3vNzF43sxfNbE5QeUREhqKgF6WBAJeqdPf1wFwAM0sGtgOP9zjsHeB97l5tZhcCy4DTgsokIjLU1Da3k5xkZKclB3aNQNcs7uZ8YJO7b+2+0d1f7PbyJWBCjPKIiAwJdc0d5GakYGaBXSNWYwSfAh46wjHXA0/2tsPMFplZmZmVVVZWDno4EZGwqmsJdi0CiEEhMLM04GPAI4c55lwiheCrve1392XuXurupQUFBcEEFREJoaDXIoDYdA1dCLzi7rt722lmJwF3ARe6+74Y5BERGTKCnnAOYtM1dAV9dAuZ2STgMeBqd98QgywiIkPKkG8RmFk2cAHw2W7bbgJw96XA14HRwJ3RgZAOdy8NMpOIyFBS19IR6FoEEHAhcPdGIh/03bct7fb9DcANQWYQERnKagNeuB70ZLGISGi1tHfS1tE1LMYIRETkKNTFYHoJUCEQEQmtWEwvASoEIiKhtX/CuSDXIgAVAhGR0KprjixTqa4hEZEEpa4hEZEEF4u1CECFQEQktA6sTqbbR0VEElNtczuZqcmkpQT7Ua1CICISUnXNwU8vASoEIiKhFYsJ50CFQEQktOpagp+CGlQIRERCKxYTzoEKgYhIaNW1qGtIRCSh7V+4PmgqBCIiIdTV5UO/RWBmM81sTbevOjP7Uo9jzMxuM7ONZvaamZ0cVB4RkaGkoa0D9+Cnl4AAVyhz9/XAXAAzSwa2A4/3OOxCYHr06zRgSfRPEZGEVtsUm3mGIHZdQ+cDm9x9a4/tFwP3ecRLQL6ZjY9RJhGR0Dow4dwwun30U8BDvWwvBsq7va6IbhMRSWg10RZBftYwKARmlgZ8DHjkGM6xyMzKzKyssrJy8MKJiIRUdVMbACOz0gK/VixaBBcCr7j77l72bQcmdns9IbrtEO6+zN1L3b20oKAgoJgiIuFRHW0RjBwOLQLgCnrvFgJ4ArgmevfQ6UCtu++MQSYRkVCraYy0CPJj0CII9EkFM8sGLgA+223bTQDuvhT4HfBhYCPQBCwMMo+IyFBR3dROdlrwU1BDwIXA3RuB0T22Le32vQM3B5lBRGQoqmlqi0lrAPRksYhIKFU3tTEyO/jxAVAhEBEJpeqm9pjcMQQqBCIioaSuIRGRBBdpEahrSEQkIXVGZx5Vi0BEJEHVNrfjHpuHyUCFQEQkdKoaYze9BKgQiIiETk3T/qeK1SIQEUlI++cZGpWtFoGISEKK5cyjoEIgIhI66hoSEUlw1U3tpCQZI9IDnQ7uABUCEZGQ2f9UsZnF5HoqBCIiIVPdGLunikGFQEQkdKqb2mI2UAwqBCIioVPT1B6zgWJQIRARCZ1h1SIws3wzW2Fm68zsLTM7o8f+PDP7tZm9amZvmpmWqhSRhObukRZBjBalgYCXqgRuBZ5y98vMLA3I6rH/ZmCtu19kZgXAejN7wN3bAs4lIhJKTW2dtHV2xbRFEFghMLM84BzgOoDoh3vPD3gHcixyj9QIoAroCCqTiEjYHXyqeHiMEUwBKoHlZrbazO4ys+wex9wOHA/sAF4HvujuXT1PZGaLzKzMzMoqKysDjCwiEl810XmGYrUWAQRbCFKAk4El7j4PaARu6XHMB4E1QBEwF7jdzHJ7nsjdl7l7qbuXFhQUBBhZRCS+Yj3PEARbCCqACndfGX29gkhh6G4h8JhHbATeAWYFmElEJNT2zzw6LLqG3H0XUG5mM6ObzgfW9jhsW3Q7ZjYWmAlsDiqTiEjYHZxwbhgMFkctBh6I3jG0GVhoZjcBuPtS4JvAvWb2OmDAV919b8CZRERCq7px/xjBMLl91N3XAKU9Ni/ttn8H8IEgM4iIDCXVTW3kpKeQmhy75331ZLGISIhUN7XF9GEyUCEQEQmV6qb2mN4xBCoEIiKhsn8tglhSIRARCZHIhHPqGhIRSVg1jeoaEhFJWO2dXdS3dsT01lFQIRARCY2aA08Vq0UgIpKQDj5VHMIWgZllm1lS9PsZZvYxM4tt0mP0ankN//DIq9RGK66ISNhUh7xF8ByQYWbFwB+Aq4F7gwoVhKrGNlasqmD97vp4RxER6dX+mUdHZYezEJi7NwGXAne6++XAicHFGnzTx44AYIMKgYiEVKi7hgCLrjd8JfDb6LbkYCIFozg/k+y0ZN5WIRCRkNrXGO4WwZeArwGPu/ubZjYVeDq4WIPPzJg2Noe39zTEO4qISK9217aQk55CVlrQE0Mfql9Xc/dngWcBooPGe939C0EGC8KMwhE8vV5LXYpIOO2qa2FcXkbMr9vfu4YeNLPc6JrDbwBrzewrwUYbfDPG5rC3oZXqaPNLRCRMdtW1hrcQACe4ex1wCfAkkYXprw4sVUA0YCwiYba7toWxueEtBKnR5wYuAZ5w93bAj/QmM8s3sxVmts7M3ooOOPc8ZoGZrTGzN83s2YHFH5gZY3MA2KBxAhEJmY7OLiobWhkXh0LQ3xGJHwNbgFeB58xsMlDXj/fdCjzl7pdFl6vM6r7TzPKBO4EPufs2Myvsd/KjMD4vg5z0FN05JCKhs7ehjc4uZ2wcuob6O1h8G3Bbt01bzezcw73HzPKAc4DroudoA3p2zn8aeMzdt0WP2dO/2EcncufQCHUNiUjo7KprAYhLi6C/g8V5ZvZ9MyuLfn0PyD7C26YAlcByM1ttZndFB5u7mwGMNLNnzGyVmV3Tx/UX7b92ZeWx3fUzozCHt3era0hEwmVXbcgLAXAPUA98IvpVByw/wntSgJOBJe4+D2gEbunlmPnAR4APAv9iZjN6nsjdl7l7qbuXFhQU9DNy76aPHcG+xjb2NbQe03lERAbT7v0tgrB2DQHHufvHu73+NzNbc4T3VAAV7r4y+noF7y4EFcA+d28EGs3sOWAOsKGfuQbswIDx7gbOGJEe1GUGpKW9k417GqiobqKiupmdtS10dkXG4s1g1rgczp1ZSGEcflMQkdjYVddCarIxOsZPFUP/C0Gzmb3X3f8PwMzOApoP9wZ332Vm5WY2093XA+cDa3sc9ivgdjNLAdKA04AfDOhvMED7byHduKeeM44bHeSl+tTU1sHzb+/lL5v2sXpbNWt31tHeefAmrMzUZFKTDYCOLqeprROA2cW5zJs4krG56RTmZpCekkR5VRNb9zWxt6GVyaOzOX58DsePz6VkTDa5GUNqgtij1tLeyfpd9eysbSE9NYmMlGSy05Mpzs9kVHYaZjbgc9Y2t7Nhdz3rdtWzYVdkTGlcXgZjczOYMDKTGWNzYj4NgAxvu2tbKMzJIClp4P9ej1V/C8FNwH3RAWCAauDafrxvMfBA9I6hzcBCM7sJwN2XuvtbZvYU8BrQBdzl7m8M6G8wQONyI3cObYjDOMGTr+/k0VcqeP7tvbR2dJGRmsRJE/K5/r1TOWlCHpNGZTFxZBa5mSkHPrzcnbd21vP0+j08vW4Pv1qznbqWjkPOW5iTzqjsNF7aXEVze+eB7flZqUwcmcXpU0dx7ZklTBh5yE1bA9LW0cXmvQ1srmxkX2Mb1Y1t1DS1k5aSRHZaMlnpKVQ3trFlXyNb9zXR2NZBfmYqeZmpjB6RTsnoLErGZDNlTDbTCkeQntL3VFV76lp4eUs1O2qa2VHbzJ76VtKSk8hMSyYjJZm2zk4aWzupb+mgvKqJjZUNB1pQPY1IT2HCyExyM1NJT0mKfKUmk5maTFZaMslJRltHFy3tXTS2dlBR00R5VTO1zQenK89JTyEpyQ7ZBjBmRBrHj8/lzOPG8L4ZBRw/Pueoio4IRFoEY3Pj00th7kd8HODgwWa5AO5eZ2ZfcvcfBpasD6WlpV5WVnZM57j0zhdITU7i559912MNgXB3fvTnjXz/jxsozs/kghPG8oETxnLKlFGkJg98baCW9k721LXS2tHJhJFZZKZFPlS7upxtVU2s21XH1n1NbIu2Fv6yeR/uzoXvGc81p0+mtGQUyd1+66isb+XZDZU0tnYcyFvX0kFlfSt76lvYuq+JTZUNh7RaALLTkmnvcto6ugBITjImjsxk8uhsRmSkUNfcTk1TO3vqW9hdd3BMJiXJmFY4ghOKcpk4MosxOekUjEhj674mfv/mLl7ZVnPINcbmZtDe1UVzWxct7Z2R4pOeTHZaCuPzMjixKI8Ti3KZOCqLts7IMQ0tHVRUN7OtqonyqiYaWjto6+yitb2Llo5Omts6aWrrpLPLSU9JIiM1mcy0SCti4qhMJozMYsbYEcwcl0tRXgZmRnNbJ7vqWti6r5GNexrYsLue1ypqWRdtMYwZkUZBTgaZqZGiNWZEOsX5mRSPzGTMiHQyUpPJSEkiOz2S+2hbKzI8nfe9Z5g1Loc7r5wfyPnNbJW7l/a2b0AzG0WfLt7vy0DMC8FgmDE2hz+s3R2Ta7k73/39eu58ZhOXnlzMdy+bc8iH8NHISE1m0uh3/3aflGSUjMmmZMyhN2ftqGnmp3/ZwoMrt/Hb13YyMiuVBTMLmTUuh2fWV7LynX309gt1XmYqhTnpFI/MZMHMQo4fn8O0whEUjEgnPyuNtJRIEWvr6KK5rZOs9OQ+C1tTWwdb9kYKyrpddby5o44XNu5lT30r3X8XmV2cy99fMIMFMwuZNDqL3IyU0HxYZqYlMyXaqlkw8+AjL7vrWnhuQyUvba6itrmdlvZOmto6eGVbNb99bScdfbRW0lOSKMrPpDg/k6L8DIrzsygemUlRXkZk+8jMo/pFQYYed2dXbQvvm3FsN8McrWOZ4i4c/3ceheljc3j45XL2NrQy5igGjN2d+1/ayoyxOZw2te9xhq4u55u/XcvyF7bw6dMm8R8Xz45L/19RfiZfu/B4Fp83nafXRbqYntlQyeOrt3NcQTafP3caH5o9/pC7FbLSkslI7d9M42kpSQeKQl+y0lI4oSiXE4pyuWhO0YHtHZ1dVDW2UdnQSn5WGsX5mUf3l4yjsbkZXF46kctLJ75rX2eXs7uuharGNlo7Omlp76K+pZ2dtS2Rrq+aFrbXNPP0+koq6w+9ky07LZmzpo3hfTMLOG9WIePzht7PRvqnvrWDprZOxsfhjiE4tkLQ/z6lkJnRbc6hoykEP3tpK1//1Ztkpibz8KLTmTMx/13HVDW28Xc/X8OzGypZeFYJX//oCXH/zXZEegoXzSniojlFdHY5lfWtjM1Nj2uulOQkCnMzhu0dUclJRlF+JkX9KHAt7Z3srosUhu3Vzawur+HZ9ZX8Ye1ukgw+NHscnz3nuF7/vcnQtjv6DEE85hmCIxQCM6un9w98A4bsryezxuUC8NyGvZx53JgBvfcvm/bxb79eyzkzCnhnbwPX//RlHvvbsw7pqinbUsXih1azr6GNb14ym6tOmxT3ItBTcpLF5X5l6VtGajKTR2czeXSka+/y0om4O5sqG1ixajsPrNzK717fxaklozj/+EJKS0bxnuK8I7bGJPzi+VQxHKEQuHtOrILEUkFOOhfNKeKnL27hM+8toTCnfz/88qombn7wFaaMyeaOT89jd10rH1/yItct/yvLrpnP6m01/HndHv6wdjfF+Zk89rkzmV2cd+QTi/TBzJhWmMMtF87i5nOP4+cvl/PgX7fxrSfXAZCRmsTFc4r5wvunD8luNYk48FTxEOwaGtK+fMEMfvf6Tu7480b+7eLZRzx+b0Mri362ivbOLpZdPZ+cjFRyMlK569pSrrxrJe///nNApKJfffpkvvyBGQlzH7/ERk5GKjecPZUbzp5KZX0rq7ZW8eyGvTy6qoLHV2/nilMn8rlzp8Wte0GO3v6nikPZNTScTRmTzSdKJ/LgX7dxw9lTmTiq73vsX9y4ly/+fA11ze38+Or5TC0YcWDfKSWjuPvaUl6rqGXBzAJOGJ8bum4gGX4KctL50OzxfGj2eBafN40f/XkjD6zcxgMrt/GRk8az8KwpzNVYwpCxq66F/KzUft+gMdgSunPxC+dPw8z44Z/e7nV/e2cX3//jBq68eyW5GSn86vNnHXLb4H5nTy/g5nOncWJRnoqAxFxRfibfuvQ9/PnvF3D1GZP537f2cMkdL3DpnS/wp7W7GcizQhIfu2rjsw7BfgldCMbnZXLtGZN5fHUFa3ccfESiqa2D5S+8w4LvPsNt//s2Hz95Ar9e/N4Dg8wiYTRpdBbfuOhE/vK18/jGRSewu66VG+4r48Jbn+fXr+6go7Mr3hGlD7vjtFbxfgN6sjgMBuPJ4u6qGts457+fpqG1g9yMFCaOymJHTTPVTe2cUjKSzy2YxrmzAl0vRyQQ7Z1dPLFmB3c+s5FNlY1MHJXJDe+dyuWlE8hKS9he4VAq/Y8/8f7jC/n2x08K7BqD9mTxcDQqO41H//ZMnt2wh/KqZsqrmygZnc3Cs0ooLRkV73giRy01OYmPz5/AJfOK+ePaXSx7bjPfeOJNfvCnDXzhvOlcc8ZkUvTkcty1d3axr7E1roP8CV8IAGaOy2HmuGF5p6wIyUl2YGC5bEsVt/7v2/z7b9byi7Jy/v3i2Zw6Rb/wxNP+aVbi2TWkXwdEEkhpySju+8yp/Pjq+dS3dPCJH/+Ff3r89QMTB0rsxXNlsv3UIhBJMGbGB08cxznTC/j+H9fzk+ffYf2uepZcNZ+CnHAs1pRI4v0MAahFIJKwMtOS+aePnMCPrpjHGztqufj2/+ON7bXxjpVw4v1UMagQiCS8i+YUseKmMwH4xI//woub9sY5UWLZXddCWkoSI7PiNxOBCoGIMLs4j1/efBYTRmaycPnLPLN+T7wjJYxddS2My82I68OogRYCM8s3sxVmts7M3jKzXpcEM7NTzKzDzC4LMo+I9K0wN4OHF53BtMIR3HhfGb9/c1e8IyWEiupmivLjOz9U0C2CW4Gn3H0WMAd4q+cBZpYMfAf4Q8BZROQIRmWn8eCNpzO7OI/PPfAKv1y9Pd6Rhr3yqiYmHsN64oMhsEIQXej+HOBuAHdvc/eaXg5dDDwKqC0qEgJ5man87PrTOLVkFH/3izXc/9LWeEcatlraO9lT38qkw0x6GQtBtgimAJXAcjNbbWZ3mdkhi+maWTHwN8CSw53IzBaZWZmZlVVWVgaXWESAyGp2yxeewvmzCvnnX77Bkmc2xTvSsFRR3QRw2NmPYyHIQpACnAwscfd5QCNwS49jfgh81d0P+zSLuy9z91J3Ly0oiM/iziKJJiM1mSVXzedjc4r4zlPruOPpjfGONOyUVzUDMHFUfBcVCvKBsgqgwt1XRl+v4N2FoBR4ODpaPgb4sJl1uPsvA8wlIv2UmpzEDz45l+Qk47u/X09+VipXnjY53rGGjW1V4WgRBFYI3H2XmZWb2Ux3Xw+cD6ztccyU/d+b2b3Ab1QERMIlOcn478tOora5nX/+5RvkZaby0ZOK4h1rWCivaiIjNYmCEfF9ojvou4YWAw+Y2WvAXOC/zOwmM7sp4OuKyCBKTU7ijk+fTOnkkfzdz9fw7AaN1Q2G8uomJozMivuCVoEWAndfE+3bP8ndL3H3andf6u5Lezn2OndfEWQeETl6mWnJ3HXtKUwrzOGmn61i1daqeEca8rZVNcf9jiHQk8UiMgB5manc95lTGZubzsLlL/PWzrojv0l65e5UVDUxcWR8B4pBhUBEBqggJ537bziNrLQUrr77r2zZ2xjvSENSbXM79a0dcR8oBhUCETkKE0Zmcf8Np9LlzlV3r2RPdCpl6b+w3DEEKgQicpSmFeZw78JTqG5s45p7/kptc3u8Iw0pB54hiPP0EqBCICLH4KQJ+Sy9ej6bKhu48b4yWto74x1pyDjYItAYgYgMcWdPL+B7n5jLy1uq+MJDq+ns8nhHGhLKq5sYmZVKTkb81iHYT4VARI7Zx+YU8Y2PnsAf1u7m6796A3cVgyMpr2oKxfgAaM1iERkk1501hd31rSx5ZhPjcjNYfP70eEcKtfKqJk4syot3DEAtAhEZRP/4wZlcOq+Y7/1xA794uTzecUKrs8vZXtOsFoGIDD9mxncuO4nKhla+9vjrFOSmc+7MwnjHCp3ddS20d3ooBopBLQIRGWSpyUksuWo+s8bl8PkHXuHNHbXxjhQ6B+4YCsGto6BCICIBGJGewj3XnUJuZiqfufdldtY2xztSqJRHC0EY5hkCFQIRCcjY3Azuue4UGls7Wbj8Zepb9MDZfuVVTZhBUb66hkRkmDt+fC53Xnkyb+9p4O9+voYuPWMAQHl1M+NzM0hLCcdHcDhSiMiwdc6MAv7lI8fzp7f2aLnLqDA9QwAqBCISA9eeWcIlc4v4/p828Mz6PfGOE3fbEqkQmFm+ma0ws3Vm9paZndFj/5Vm9pqZvW5mL5rZnCDziEh8mBnfuvQkZo7N4YsPrzkwWJqIGlo72FPfypQx2fGOckDQLYJbgafcfRYwB3irx/53gPe5+3uAbwLLAs4jInGSmZbM0qvm0+XOZ3+2isbWjnhHiovNlQ0AHFeQAIXAzPKAc4C7Ady9zd1ruh/j7i+6e3X05UvAhKDyiEj8lYzJ5kdXzGPdrrqEHTzedKAQjIhzkoOCbBFMASqB5Wa22szuMrPDlcDrgSd722Fmi8yszMzKKiu1aLbIULZgZiH/Ep2g7r9/vz7ecWJuc2UjyUnGpNGJMUaQApwMLHH3eUAjcEtvB5rZuUQKwVd72+/uy9y91N1LCwoKgsorIjFy3ZklXHX6JJY+u4lHyhJrTqJNlQ1MGpVFekpyvKMcEGQhqAAq3H1l9PUKIoXhEGZ2EnAXcLG77wswj4iEhJnxjYtO5OzpY/h/j7/Oq+U1R37TMLFpT2OoxgcgwELg7ruAcjObGd10PrC2+zFmNgl4DLja3TcElUVEwic1OYkfXTGPwpwMPv/QK9QlwJPHnV3OO/samRqi8QEI/q6hxcADZvYaMBf4LzO7ycxuiu7/OjAauNPM1phZWcB5RCRE8rPSuO2KeeyoaeGWR18b9gvabK9upq2jK3QtgkCnoXb3NUBpj81Lu+2/AbghyAwiEm7zJ4/kKx+cybefXMf9K7dx9emT4x0pMGG8Ywj0ZLGIhMCis6eyYGYB3/zNWt7YPnynrVYhEBHpQ1KS8b3L5zAqK42bHxy+4wWbKhsYlZ3GyOy0eEc5hAqBiITC6BHp3P7peVRUN/OPjwzP8YJNlY1MDdHUEvupEIhIaJSWjOKWD83iqTd3cc8LW+IdZ9BtrmwIXbcQqBCISMjccPYULjhhLN/63Vus2lp95DcMETVNbextaOO4QrUIREQOy8z4n8vnMD4/g88/+ApVjW3xjjQoNlU2AuEbKAYVAhEJobzMVJZcOZ99jW188eHVdA6Dyek2h/SOIVAhEJGQml2cx79edCLPv72X2/889Fc221TZSFpyEhNGhmOd4u5UCEQktK44dSKXzivmh/+7geffHtozD2+qbGDy6CxSksP3sRu+RCIiUWbGf/zNbKYXjuCLD69hZ21zvCMdtbDeMQQqBCIScllpKSy5aj6t7Z3c/MArtHd2xTvSgLV3drF1X1Mo7xgCFQIRGQKOKxjBtz9+Eq9sq+HbT66Ld5wB27qvkY4uV4tARORYXDSniOvOLOHu/3uHJ1/fGe84A7KmPDJ/0kkT8uKcpHcqBCIyZPy/Dx/P3In5fGXFawduxxwKVm+rJicjhalj1CIQETkmaSlJ3HHlyaQmG397/ys0tXXEO1K/rN5Ww9yJ+SQlWbyj9EqFQESGlOL8TG791Dw27Knnnx9/I/ST0zW1dbB+dz3zJubHO0qfAi0EZpZvZivMbJ2ZvWVmZ/TYb2Z2m5ltNLPXzOxdaxqLiPR0zowCvnT+DB5bvZ0H/7ot3nEO6/WKWjq7nLmTErQQALcCT7n7LGAO8FaP/RcC06Nfi4AlAecRkWFi8XnTWDCzgH97Yi2rtlbFO06fVpfXADB34sg4J+lbYIXAzPKAc4C7Ady9zd1rehx2MXCfR7wE5JvZ+KAyicjwkZRk/PCTcynKz+DG+1axZW9jvCP1avW2akpGZzEqZDo+1esAAAxeSURBVIvRdBdki2AKUAksN7PVZnaXmfV8mqIYKO/2uiK6TUTkiPKz0li+8FTcnYX3vhy6mUrdndXbapg3KbytAQi2EKQAJwNL3H0e0AjccjQnMrNFZlZmZmWVlUN7vhERGVxTxmTzk2tK2V7TzKL7ymhp74x3pAN21rawp76VuSEeKIZgC0EFUOHuK6OvVxApDN1tByZ2ez0huu0Q7r7M3UvdvbSgoCCQsCIydJWWjOL7n5hD2dZqbnk0PMtcrt4W6Q2fF+KBYgiwELj7LqDczGZGN50PrO1x2BPANdG7h04Hat19aD0yKCKh8NGTivj7C2bwyzU7+PFzm+MdB4iMD6SnJDFrXG68oxxWSsDnXww8YGZpwGZgoZndBODuS4HfAR8GNgJNwMKA84jIMPb586axblc933lqHTPH5nDurMK45llTXsPs4jzSUsL9yFaghcDd1wClPTYv7bbfgZuDzCAiicPM+O7lJ7FlXyNfeGg1j998JtMKc+KSpa2ji9e313L16ZPjcv2BCHeZEhEZoKy0FJZdU0p6ahLX3vMyFdVNccmxblcdrR1dob9jCFQIRGQYKs7P5N6Fp1Lf0s6nf7IyLgvaPLchcodj2AeKQYVARIap2cV53Hf9aVQ1tvHpn6xkT11LzK7d3tnF/S9t473TxlCUH741intSIRCRYWvuxHx++plT2FPXwhU/eYm9Da0xue7v39zFrroWFp5VEpPrHSsVAhEZ1uZPHsU9153C9ppmrr77r9Q0Bf/08b0vbGHy6CzOnRnfu5b6S4VARIa906aO5ifXlLJpTwPXLn+Z+pb2wK71ekUtZVurufaMktCuP9CTCoGIJISzpxdwx5Un8+b2Wq6/t4y6gIrB8hffITstmctKJwRy/iCoEIhIwrjghLH84JNzWbWtmotvf4H1u+oH9fyV9a385tWdXDZ/ArkZqYN67iCpEIhIQrloThEP3Xg6Da0dXHLHC/xqzbumNztq9774Dm2dXVx7ZsmgnTMWVAhEJOGcOmUUv138XmYX5/LFh9fw2Z+VsW5X3TGds2xLFUuf3czFc4uYWhDORer7okIgIgmpMDeDB288nS9fMIMXN+7jwlufZ/FDq4+qu6iqsY3FD61mwshMvnnJ7ADSBivoSedEREIrNTmJL5w/nWvOmMyy5zZz74tb+PWrOzilZCRXnT6ZD80eR3pK8mHP0dXl/P0v1rCvoY3HPnfmkBob2M/CMm93f5WWlnpZWVm8Y4jIMFTd2MYjq8p5YOU2tu5rIicjhXNmFHDezELOmVFAQU76IceXVzVx31+28JPn3+GbF5/I1WeUxCV3f5jZKnfvOQloZJ8KgYjIobq6nBc27eXXr+7g6fWVVNZHnkjOy0xl0qgsxuZm8NbOOrbXROYwumRuET/45FzMwvvcwOEKgbqGRER6SEoyzp5ewNnTC+jqctburOOlzfvYuq+JrVVNbN3XyEkT8lh0zlTOOG400wtHhLoIHIkKgYjIYSQlGbOL85hdnBfvKIHRXUMiIgku0BaBmW0B6oFOoKNn/5SZ5QH3A5OiWf7H3ZcHmUlERA4Vi66hc919bx/7bgbWuvtFZlYArDezB9w9+OkBRUQEiH/XkAM5FhllGQFUAR3xjSQikliCLgQO/MHMVpnZol723w4cD+wAXge+6O5dPQ8ys0VmVmZmZZWVlcEmFhFJMEEXgve6+8nAhcDNZnZOj/0fBNYARcBc4HYzy+15Endf5u6l7l5aUFAQcGQRkcQSaCFw9+3RP/cAjwOn9jhkIfCYR2wE3gFmBZlJREQOFVghMLNsM8vZ/z3wAeCNHodtA86PHjMWmAlsDiqTiIi8W2BTTJjZVCKtAIjcnfSgu/+nmd0E4O5LzawIuBcYDxjwbXe//wjnrQS2Rl/mAbWH+b7ntlSgrzuY+tL9HP3Z13NbfzPu/3PMADPGKt/+bfoZhivfUMgY9nzHkvFw28L2M5zs7r33rbv7kP0Clh3u+57bgLJjuUZ/9vXc1t+M3f4cUMZY5dPPMJz5hkLGsOc7loxHyBqqn+HhvuJ9++ix+vURvu9r/9Feoz/7em7rb8aw5zvStQ5HP8MjX+dwjvS+sGcMe76+9vcn45G2DUTQP8M+DbnZR4+FmZV5H7PvhUXYM4Y9H4Q/Y9jzQfgzhj0fDI2M+w31FsFALYt3gH4Ie8aw54PwZwx7Pgh/xrDng6GREUiwFoGIiLxborUIRESkBxUCEZEEp0IgIpLgVAiizOxsM1tqZneZ2YvxztMbM0sys/80sx+Z2bXxztOTmS0ws+ejP8cF8c7Tm+gT72Vm9tF4Z+mNmR0f/fmtMLO/jXee3pjZJWb2EzP7uZl9IN55ejKzqWZ2t5mtiHeW/aL/7n4a/bldGe88PQ2LQmBm95jZHjN7o8f2D5nZejPbaGa3HO4c7v68u98E/Ab4aRgzAhcDE4B2oCKE+RxoADJCmg/gq8AvBjPbYGZ097ei/w4/AZwV0oy/dPcbgZuAT4Yw32Z3v34wc/VmgFkvBVZEf24fCzrbgA3kybewfgHnACcDb3TblgxsAqYCacCrwAnAe4h82Hf/Kuz2vl8AOWHMCNwCfDb63hUhzJcUfd9Y4IEQ5rsA+BRwHfDRMP43jr7nY8CTwKfDmjH6vu8BJ4c436D+P3KMWb8GzI0e82CQuY7ma1gsXu/uz5lZSY/NpwIb3X0zgJk9DFzs7t8Ceu0WMLNJQK2714cxo5lVAPtXb+sMW75uqoH0sOWLdldlE/kfs9nMfue9rH8Rz4zR8zwBPGFmvwUeHKx8g5XRzAz4NvCku78StnyxMpCsRFrIE4hMux+6nphhUQj6UAyUd3tdAZx2hPdcD8RyzeSBZnwM+JGZnQ08F2SwqAHlM7NLiawxkU9k0aGgDSifu/8TgJldB+wdzCJwGAP9GS4g0o2QDvwu0GQHDfTf4WLg/UCemU1z96VBhmPgP8PRwH8C88zsa9GCESt9Zb2NyHorH+Hop6AIzHAuBAPm7t+Id4bDcfcmIsUqlNz9MSLFKtTc/d54Z+iLuz8DPBPnGIfl7rcR+WALJXffR2T8IjTcvZHI+iuhFLomyiDaDkzs9npCdFuYhD2j8h07ZTx2Yc/X3VDKesBwLgQvA9PNbIqZpREZJHwizpl6CntG5Tt2ynjswp6vu6GU9aB4j1YP0uj9Q8BODt5WeX10+4eBDURG8f9JGZVPGcOdMez5hmrWI31p0jkRkQQ3nLuGRESkH1QIREQSnAqBiEiCUyEQEUlwKgQiIglOhUBEJMGpEMiwYGYNMb7eoKxZYZE1HGrNbI2ZrTOz/+nHey4xsxMG4/oioEIg0iszO+w8XO5+5iBe7nl3nwvMAz5qZkdah+ASIjOoigwKFQIZtszsODN7ysxWWWTltFnR7ReZ2UozW21mfzKzsdHt/2pmPzOzF4CfRV/fY2bPmNlmM/tCt3M3RP9cEN2/Ivob/QPRaZoxsw9Ht60ys9vM7DeHy+vuzUSmKS6Ovv9GM3vZzF41s0fNLMvMziSyXsF3o62I4/r6e4r0lwqBDGfLgMXuPh/4B+DO6Pb/A05393nAw8A/dnvPCcD73f2K6OtZRKbWPhX4hpml9nKdecCXou+dCpxlZhnAj4ELo9cvOFJYMxsJTOfgFOOPufsp7j4HeIvIFAYvEpm75ivuPtfdNx3m7ynSL5qGWoYlMxsBnAk8Ev0FHQ4uljMB+LmZjSeyitQ73d76RPQ38/1+6+6tQKuZ7SGy+lrPZTj/6u4V0euuAUqILNm52d33n/shYFEfcc82s1eJFIEfuvuu6PbZZvYfRNZ3GAH8foB/T5F+USGQ4SoJqIn2vff0I+D77v5EdCGYf+22r7HHsa3dvu+k9/9n+nPM4Tzv7h81synAS2b2C3dfA9wLXOLur0YX01nQy3sP9/cU6Rd1Dcmw5O51wDtmdjlEllc0sznR3XkcnCP+2oAirAemdlvK8IiLvEdbD98GvhrdlAPsjHZHXdnt0ProviP9PUX6RYVAhossM6vo9vVlIh+e10e7Xd4ksnYsRFoAj5jZKmBvEGGi3UufA56KXqceqO3HW5cC50QLyL8AK4EXgHXdjnkY+Ep0sPs4+v57ivSLpqEWCYiZjXD3huhdRHcAb7v7D+KdS6QntQhEgnNjdPD4TSLdUT+Ocx6RXqlFICKS4NQiEBFJcCoEIiIJToVARCTBqRCIiCQ4FQIRkQSnQiAikuD+P9ktaRMRbXzZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.171446</td>\n",
       "      <td>3.774947</td>\n",
       "      <td>43.595184</td>\n",
       "      <td>01:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.581508</td>\n",
       "      <td>3.392662</td>\n",
       "      <td>29.745029</td>\n",
       "      <td>01:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.233801</td>\n",
       "      <td>3.269958</td>\n",
       "      <td>26.310234</td>\n",
       "      <td>01:24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lang_learn.fine_tune(2, 4e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang_learn.save_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = DataBlock(blocks=(TextBlock.from_df(text_cols=['HashTag', 'Text'], is_lm=False, vocab=dls_lm.vocab), CategoryBlock),\n",
    "               get_x=ColReader('text'),\n",
    "               get_y=ColReader('Label'),\n",
    "               splitter=ColSplitter(col='is_valid')\n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  return array(a, dtype, copy=False, order=order)\n"
     ]
    }
   ],
   "source": [
    "dls_cls = db.dataloaders(train_val, bs=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = text_classifier_learner(dls_cls, arch=AWD_LSTM, metrics=[accuracy, F1Score(), Precision(), Recall()], loss_func=LabelSmoothingCrossEntropyFlat())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<fastai.text.learner.TextLearner at 0x7fba35afa4d0>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.load_encoder('fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.04365158379077912, lr_steep=0.0008317637839354575)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wVVfr48c+TRkgIoSTUIAQIVYoSsaAIVnQtrG1Bv7ZF7PrSXVFw13VX15/uuru6KjbsFRUbtsUGKlUCgkBoIXQCCQHSIP35/XEneIkpN5DJ5CbP+/W6L+49c2bmmRjvk3POzDmiqhhjjDGBCvE6AGOMMcHFEocxxpg6scRhjDGmTixxGGOMqRNLHMYYY+rEEocxxpg6CfM6gIYQFxenPXr08DoMY4wJKkuWLNmtqvGVy5tF4ujRowcpKSleh2GMMUFFRDZXVW5dVcYYY+rEEocxxpg6scRhjDGmTixxGGOMqRNLHMYYY+rEEocxxpg6scRxmLbvO0BmXqHXYRhjTIOzxHGYbn5jCZc8s4D9xaVeh2KMMQ3KEsdhKCtXVu/MY8ue/fzzf2u9DscYYxqUJY7DsG3vfopLy+napiWvzN/EovRsr0MyxpgGY4njMKRl5gPwj4sHc1S7KO5+/2frsjLGNBuWOA7DeidxDEqI5R8XD2ZztnVZGWOaD1cTh4iMEZG1IpImIpOr2P6YiCxzXutEZJ/ftjK/bTP9yhNFZJFzzHdEJMLNa6hKWmY+HWJaENsynBN7tefqE7vzyvxNbNxd0NChGGNMg3MtcYhIKDAVOAcYAIwXkQH+dVT1TlUdqqpDgSeBD/w2H6jYpqoX+JX/A3hMVXsDe4EJbl1DddZn5tO7Q6uDn68ZkQjA/A27GzqUBjEvbTe7cu3WY2OMj5stjuFAmqqmq2oxMB24sIb644G3azqgiAhwGjDDKXoVGFsPsQZMVdlQKXH0aB9Fh5gW/LhxT0OG0iB+3LiHK15YxPlPziV1R67X4RhjGgE3E0dXYKvf521O2a+ISHcgEfjWrzhSRFJEZKGIVCSH9sA+Va0Yia7pmNc7+6dkZWUdyXUcYmduIflFpST5JQ4RYXhiOxal70FV6+1cXispK+fPH62gc2wkoSHC755b0GRbVcaYwDWWwfFxwAxVLfMr666qycDlwOMi0qsuB1TV51U1WVWT4+N/tYDVYau4o6qXX+IAOD6xHTtzC9m650C9nctrL83dyLpd+fztgoG8f9NJdIqN5JqXFvNeylZKysq9Ds8Y4xE3E8d2oJvf5wSnrCrjqNRNparbnX/TgTnAMUA20EZEKlYurOmYrqhIHEkdYg4pP75newAWbQz8mY70rHye+24DhSVltVeuxq7cQsY9v4AnvlnP3oLiwz5OZdv3HeDxr9dzRv8OnDWwE13atGTGjScxpFssk2b8zPCHvmbKByuYl7ab8vKm08oyxtTOzcSxGEhy7oKKwJccZlauJCL9gLbAAr+ytiLSwnkfB4wAUtXXDzQbuMSpejXwsYvX8CvrM/OJbRlOXKtDb+bqHd+KtlHhdRrnuH/mKh7+Yg2XPruArXv2H1Y8r87fxML0Pfznq3Wc+Mg3/OXjlWyqh7u7HvhkFYpy//kDD5bFRoXz1sQTeP7KYZySFM/Hy7ZzxQuLuPnNpUeU/IwxwcW1xOGMQ9wKzAJWA++q6ioReUBE/O+SGgdM10MHB/oDKSKyHF+ieERVU51t9wB/EJE0fGMeL7p1DVVJcwbGfeP0vwgJEY7r0Y4fNwWWONbuzOOH9bs5a0BHNmUXcP5Tc5mzNrNOsRSWlDF98VbOGtCRL+8cyfmDu/D2j1sY9a85jHt+AR/9tP2wvtBnr8lk1qpd3H56Et3aRR2yLTw0hLMGduKJ8cew9L4zmXxOP2al7mT8tIVk5xfV+VzGmODj6hiHqn6uqn1UtZeqPuSU/UVVZ/rV+auqTq6033xVHaSqQ5x/X/Tblq6qw1W1t6peqqoN+m2Vlpl/yMC4v+GJ7dicvZ+dOb/curp1z36mzk771ZjAi3PTiQwP4R8XD+aTW0+mU+tIrn1lMZ+vyAg4lk9/zmBPQTFXn9SDPh1jePTSIcy75zQmnd2XjJxC7nhnGSc+/A1rdgZ+N1RZufL/Pl9NYlw0153cs8a6keGh3HhqL5654lhSd+Ry0TPz7VkWY5qBxjI4HhSy84vYU1B8yK24/k6oNM6hqvzh3WU8Omst//lq3cF6WXlFfPTTDi4ZlkDb6Ah6xEXz4c0j6Nsxhn99uZayAMYMVJVX52+id4dWnNSr/cHyDq0juWV0b2b/cRRvTTye0JAQ7nxnOcWlgQ1mv790G+sz85l0dl8iwgL79RhzdGfemngCeYWlXD5tIflFNv2KMU2ZJY46qBgYry5x9O/cmlYtwg6Oc7y/dDuLN+2lT8dWPPvdBuan+W5lfX3hZkrKy/m98+AgQMuIUG4Z3Zv0rAK+XLWz1lh+2rqPFdtzuPrE7r/qNgNf19lJveJ45KJBrM7I5Ylv1td6zMKSMh77ah1DurXhnKM71Vrf37DubZl2VTIZOYU8GcC5jDHByxJHHaRl1Zw4QkOE5B5tWbRxDzn7S3j489Uce1QbPrx5BD3jornjnWXs2HeANxZu5vR+HekZf+hxzh3UmR7to3h6zoZanwd5bf4mYlqEcdGxCTXWO2NARy4dlsDTc9L4acveGuu+On8TGTmFTDmnX5XJqDbDurflsuQEXpy7kbTMvDrvb4wJDpY46mD9rnxahofSJbZltXWOT2xPWmY+9364gr37i3lw7NFEtwjjifHHsG9/CWOnzmNPQTHXnZL4q31DQ4QbTu3Fiu05zEur/rbezLxCPluRwSXJCUS3CKu2XoW/nD+AzrEt+eO7y9mcXcDrCzZxxQsLGf7Q1zz8xWoycwvJ2V/C1NlpjO4bf7DL7XDcM6YfURGh/OXjVQE9DFlQVMqi9GxLNMYEkdq/dcxBG7J8d1SFhFT/1/jwxHYAfLYig2tO6sHALrEADOwSyz3n9OPBT1M5umtrjnfqVXbRsV15/Ot1PD0njZOT4qqs89aiLZSUKVee0D2guGMiw3n00sFcPm0Rpz46B4Ce8dEMTohl2vfpvDzXN1aSV1TK3WP6BXTM6rRv1YJJY/px30cr+fTnDM4f0oXM3EI+/TmD9N35qILi6xZL3ZHLul15VAzpDOvelvHDj+I3gzqTW1jCpt0FbNmznwFdWh/8ORpjvGeJow7SMvNr/Wt8UNdYIsNDiIkM5w9n9Tlk2+9H9KCwpIxTkuKq7QpqERbKdSf35KHPV7Ns6z6GdmtzyPa9BcW8OHcjZ/Tv8Kuurpqc1CuOhy8axO68IsYc3Ymkjr4HGDdnFzDth3TeS9nGZcO60b9z64CPWZ3Lhx/FO4u38OCnqUxfvIUFG7IpV2gbFU5oiABCeKjQt1MMZw/sxOCEWDbuLuCtRVu4673l3PXe8kOOJwKXHJvApDF96RATCcCW7P3MXpvJvv0lB+uFhwntoyNoH92C+JgWDOzSmrBQa1QbU9+kKc2tVJ3k5GRNSUk5omPkFZYw6K9fMunsvtwyuneNdT9etp2ubVqS3KPqVkVt8otKGfHItxzXwzfg7J9k/vbJKl6dv4lZd4w8+OVfH/YXlxIRGlJvX7RLt+zlsmcX0K1dFOcP6cIFQ7pUOzZUQVVZtHEPP6zPolPrSHrERdM5tiXvpWzlpXkbaREWyvlDOrN08z7W7qq9a+vEnu159v+GERsVXi/XZExzIyJLnKmfDi23xBGYn7bs5bdPz+e5K4dx9sC63XF0OKbOTuPRWWv52wUDufqkHgBs2l3AGf/5jkuTu/HwRYNcj+FI5RaWENMi7LAG2ivbuLuAhz5LZc7aLJJ7tOXMAZ04o38HurX95QHFotJy9uwvJju/iJ+27OOhz1aT0K4lL19zHN3bRx9xDMY0N9UlDuuqCtDqDN9fuP07HXlXTiBuOrUXP23ZxwOfptIzPppTkuL556w1RISFcOeZSQ0Sw5FqHVl/f+knxkXzwtXHUV6u1Y4xtYwIpWtES7q2acnghDb079yaG15PYezUeTx/VTLHHWYL0BhzKOsADtDKHTnERIbRrV31d1TVp5AQ4fFxQ0nq0Ipb3lzKjCXb+HzFTq4f2fNgP39zVNONCZUNT2zHhzePoG1UBOOeX8g//rfG5tQyph5Y4gjQqh25DOzSul66XQLVqkUY065KJjw0hLveW058TAsmnlLzNCDmUD3iovnwlhFcfGxXnpmzgXOf+IGUAOcTM8ZUzRJHAErLylmTkevJLaHd2kXx7JXDiIkM495z+wX03IY5VGzLcP55yRBenzCc4tJyLn1uAZc+O5/nvtvABuehTmNM4GxwPABrd+Zx9uPf89jvhvDbY2p+UtstpWXldmtpPSgoKuXFuRv538qdpGb4Jn9sFx1B++gI2kZFEB/TgitOOIqTelX9DI0xzYkNjh+BVTtyADx9CM2SRv2IbhHG7acncfvpSWzfd4BvVu9idUYe+/YXs6egmEUb9/DZigzOHNCRe8/tT2Kc3Y1lTGWWOAKwcnsukeEh9LQvkSala5uWXHVij0PKCkvKeHHuRp6encaZ//mOSWf35YZT67RqsTFNnv0ZG4BVO3Lo18meQm4OIsN9sxTPnjSK0/p14JH/rbHBdGMqsW/CWpSXK6nOHVWm+egQE8ljvxtKl9iW3D3jZ7uN1xg/ljhqsXXvfvKKSjm6q02y19xEtwjjHxcPJn13AY99va72HYxpJixx1GLVDt+dN9biaJ5OTopj3HHdmPZ9Osu37vM6HGMaBVcTh4iMEZG1IpImIpOr2P6YiCxzXutEZJ9TPlREFojIKhH5WUR+57fPKyKy0W+/oW5ew8rtOYSFCH3qcUJBE1zu/U1/OsREMmnGcuuyMgYXE4eIhAJTgXOAAcB4ERngX0dV71TVoao6FHgS+MDZtB+4SlUHAmOAx0XEf37xSRX7qeoyt64BfC2O3h1aERke6uZpTCPWOjKchy8exPrMfK54YRH79hd7HZIxnnKzxTEcSFPVdFUtBqYDF9ZQfzzwNoCqrlPV9c77HUAmEO9irFVSVVbtyLHxDcPovh14avyxrNiWw8XPzGfb3v1eh2SMZ9xMHF2BrX6ftzllvyIi3YFE4Nsqtg0HIoANfsUPOV1Yj4lIi/oL+VCZeUXszi+28Q0DwG8Gd+a1CcPJyivioqfnk+qMfxnT3DSWwfFxwAxVPaQDWUQ6A68D16pquVM8BegHHAe0A+6p6oAicr2IpIhISlZW1mEFVfHEuLU4TIUTerZnxk0nERoiXPXSj2zfd8DrkIxpcG4mju1AN7/PCU5ZVcbhdFNVEJHWwGfAn1R1YUW5qmaoTxHwMr4usV9R1edVNVlVk+PjD6+Xa+X2XESol+VUTdPRp2MMr08YTlFpGRNeWUx+UanXIRnToNxMHIuBJBFJFJEIfMlhZuVKItIPaAss8CuLAD4EXlPVGZXqd3b+FWAssNKtC1i1I4fE9tG0shlpTSW9O8Qw9fJjWZ+Zzx3Tf6KsvOlPFmpMBdcSh6qWArcCs4DVwLuqukpEHhCRC/yqjgOm66HT9F4GjASuqeK22zdFZAWwAogD/u7WNVw4tCs32jxFphoj+8Rz//kD+Hp1Jg9+mkpmXiHNYbZpY2xadWOO0P0fr+TVBZsBaBsVTt9OMdx2WhIjetvU7Ca4VTetuiUOY46QqvLjxj2kZuSydmce36/LorhMmX3XqcTU47rrxjQ0W4/DGJeICMf3bM/xPdsDsHzrPi6cOo+pszcw+Zx+HkdnTP1rLLfjGtNkDOnWhouPTeCluRvZnF3gdTjG1DtLHMa44O4xfQkLFR76bLXXoRhT7yxxGOOCjq0juWV0b75M3cW8tN1eh2NMvbLEYYxLJpycSELbltz38UqW2ZTspgmxxGGMSyLDQ3not4PIzC1i7NR5XPzMfD5fkWEPC5qgZ4nDGBed2ieeBVNO4y/nDSArr4ib31zKHe8so9yShwliljiMcVlMZDi/PzmR2XeN4g9n9uGT5TtsKVoT1Ow5DmMaSGiIcNtpvdmx7wBPfptG9/bRXDIsweuwjKkzSxzGNCAR4cGxR7N1736mfPAzXdu05MRe7b0Oy5g6sa4qYxpYeGgIT18xjO7to7nxjSVsyMr3OiRj6sQShzEeiG0ZzsvXHEdYiPD7Vxazp8DWMTfBwxKHMR7p1i6K569KJiOnkBteT6GotKz2nYxpBCxxGOOhYd3b8u9Lh7B4014mv7/C1vMwQcEGx43x2PlDurBpdwH//mod7aIj+PNv+uNb4NKYxskShzGNwK2n9Sa7oJgX527kQEkZf7/waEJCLHmYxskShzGNgIhw//kDaBkRyjNzNnCguIxHLxlMWKj1JpvGxxKHMY2EiHDPmH60ahHGo7PWoqo8Pu4Yr8My5lfszxljGplbRvfmttN689GyHSzZvMfrcIz5FVcTh4iMEZG1IpImIpOr2P6YiCxzXutEZJ/ftqtFZL3zutqvfJiIrHCO+YTYKKJpgm4a1Yv20RE8/vV6r0Mx5ldcSxwiEgpMBc4BBgDjRWSAfx1VvVNVh6rqUOBJ4ANn33bA/cDxwHDgfhFp6+z2DDARSHJeY9y6BmO8EhURxg2n9uSH9but1WEaHTdbHMOBNFVNV9ViYDpwYQ31xwNvO+/PBr5S1T2quhf4ChgjIp2B1qq6UH03vL8GjHXvEozxzv+d0N1aHaZRcjNxdAW2+n3e5pT9ioh0BxKBb2vZt6vzPpBjXi8iKSKSkpWVdVgXYIyXDm117PU6HGMOaiyD4+OAGapab3MuqOrzqpqsqsnx8fH1dVhjGlRFq+O/31irwzQebiaO7UA3v88JTllVxvFLN1VN+2533gdyTGOCXlREGNeP7Mn367JYusVaHaZxcDNxLAaSRCRRRCLwJYeZlSuJSD+gLbDAr3gWcJaItHUGxc8CZqlqBpArIic4d1NdBXzs4jUY47krT+xOu+gInvo2zetQjAFcTByqWgrcii8JrAbeVdVVIvKAiFzgV3UcMF39ZndT1T3Ag/iSz2LgAacM4GbgBSAN2AB84dY1GNMYREWEMeHkRL5dk8nK7Tleh2MM0hxm40xOTtaUlBSvwzDmsOUWljDikW85uXccz/zfMK/DMc2EiCxR1eTK5Y1lcNwYU4PWkeFce1IPvli5k3W78rwOxzRzljiMCRLXjkgkKiKUp2fbWIfxliUOY4JE2+gIrjyhOzOX72DT7gKvwzHNmCUOY4LIhFMSCQ8N4ek51uow3rHEYUwQ6RATybjjuvHhT9vZse+A1+GYZsoShzFBZuLInqjCtB/SvQ7FNFOWOIwJMgltoxh7TFfe/nEL2flFXodjmiFLHMYEoRtP7UVRaTkvz9vkdSimGbLEYUwQ6t2hFWMGduLVBZvIKyzxOhzTzFjiMCZI3TyqN3mFpbyxcIvXoZhmxhKHMUFqUEIspyTF8eLcdApL6m1FAmNqZYnDmCB286je7M4v5v2l22qvbEw9scRhTBA7oWc7BnWN5aW5Gykvb/oTlprGwRKHMUFMRJhwciIbsgr4bp0tkWwahiUOY4LcuYM606l1JC/O3eh1KKaZsMRhTJCLCAvhqpO6MzdtN2t25nodjmkGLHEY0wRcPvwoWoaH8uIP1uow7rPEYUwT0CYqgkuGJfDxsh1k5dk0JMZdljiMaSKuHdGD4rJyXplvrQ7jLlcTh4iMEZG1IpImIpOrqXOZiKSKyCoRecspGy0iy/xehSIy1tn2iohs9Ns21M1rMCZY9IxvxflDuvD0nA18snyH1+GYJiwskEoiEg0cUNVyEekD9AO+UNVqJ8kRkVBgKnAmsA1YLCIzVTXVr04SMAUYoap7RaQDgKrOBoY6ddoBacCXfoefpKoz6nCdxjQL/7x4MLtyCrnznWVEtwjltH4dvQ7JNEGBtji+ByJFpCu+L/ArgVdq2Wc4kKaq6apaDEwHLqxUZyIwVVX3AqhqZhXHuQRfktofYKzGNFstI0J54Zpk+nWO4aY3lrJgQ7bXIZkmKNDEIc4X90XA06p6KTCwln26Alv9Pm9zyvz1AfqIyDwRWSgiY6o4zjjg7UplD4nIzyLymIi0CPAajGkWWkeG89rvj+eodlFc9+pitu6xv7lM/Qo4cYjIicAVwGdOWWg9nD8MSAJGAeOBaSLSxu+knYFBwCy/fabg6yo7DmgH3FNNwNeLSIqIpGRl2RO1pnlpFx3By9ceR0mZ8ux3G7wOxzQxgSaOO/B9YX+oqqtEpCcwu5Z9tgPd/D4nOGX+tgEzVbVEVTcC6/AlkgqXOec8OJaiqhnqUwS8jK9L7FdU9XlVTVbV5Pj4+AAu0ZimJaFtFBcPS+C9lG3syi30OhzThASUOFT1O1W9QFX/ISIhwG5Vvb2W3RYDSSKSKCIR+LqcZlaq8xG+1gYiEoev68p/IeXxVOqmclohiIgAY4GVgVyDMc3Rjaf2pLS8nGnf2/rkpv4ElDhE5C0Rae3cXbUSSBWRSTXto6qlwK34uplWA+86rZUHROQCp9osIFtEUvG1YCaparZzzh74WizfVTr0myKyAlgBxAF/D+QajGmOureP5oIhXXhz0Rb2FhR7HY5pIkS19qmYRWSZqg4VkSuAY4HJwBJVHex2gPUhOTlZU1JSvA7DGE+s25XHWY99z+2n9eYPZ/X1OhwTRERkiaomVy4PdIwjXETC8XUNzXTGHGzyf2OCQJ+OMZw9sCOvzLf1yU39CDRxPAdsAqKB70WkO2DTcBoTJG4Z3ZvcwlKe+87GOsyRC3Rw/AlV7aqq5zp3NG0GRrscmzGmngxOaMPYoV14anYaT327nkC6qI2pTqBTjsQC9wMjnaLvgAeAHJfiMsbUs0cvHYKI8K8v15FzoIR7z+2P7+ZEY+omoMQBvITvbqrLnM9X4nuG4iI3gjLG1L/w0BD+fekQYiLDmPbDRvIKS3n4okGWPEydBZo4eqnqxX6f/yYiy9wIyBjjnpAQ4W8XDKRVizCenrOBPh1j+P3JiV6HZYJMoIPjB0Tk5IoPIjICOOBOSMYYN4kIk87uy+n9OvDIF2tI3WH3uZi6CTRx3AhMFZFNIrIJeAq4wbWojDGuEhH+eclgYqPCuX36TxwoLvM6JBNEAr2rarmqDgEGA4NV9RjgNFcjM8a4qn2rFvznsiGkZebz989Sa9/BGEedVgBU1VxVrWjX/sGFeIwxDeiUpHhuGNmTNxdt4avUXV6HY4LEkSwda7diGNME/PGsvvTrFMODn6ZSXFrudTimnizdspdb3lrKtr31vx7LkSQOe4LImCYgIiyEKef2Z8ue/by5aLPX4Zh6snZnHp/9nEGIC7db15g4RCRPRHKreOUBXeo9GmOMJ0YmxTGid3ue/DaNXJvPqknI2HeAEIEOMfW/SGqNiUNVY1S1dRWvGFUN9BkQY0wjJyJMOac/ewqKec5WDGwSduQU0iEmkrDQI+lYqlr9H9EYE5SO7hrLhUO78OLcjezMsRUDg93OnEI6t4l05diWOIwxB911Vl/Ky+Gxr9Z5HYo5QjtyDtAltqUrx7bEYYw5qFu7KK46sTvvLtnKjxv3eB2OOUyqSsa+QjrFWovDGNMA7jyzD93bRXHnO8vIOWAD5cEo90ApB0rK6GyJwxjTEKJbhPH4uGPYmVvIXz5e6XU45jDsyPFNJdjZuqqMMQ1laLc23HF6Eh8v28FHP233OhxTRxkViSMYB8dFZIyIrBWRNBGZXE2dy0QkVURWichbfuVlIrLMec30K08UkUXOMd8RkQg3r8GY5urm0b1J7t6W+z5aydY99f/0sXFPhnNXXNANjotIKDAVOAcYAIwXkQGV6iQBU4ARqjoQuMNv8wFVHeq8LvAr/wfwmKr2BvYCE9y6BmOas9AQ4bHfDUWBSTOWU15uk0UEi4x9hYSGCPEuPPwH7rY4hgNpqpquqsXAdODCSnUmAlNVdS+AqmbWdEDxLVV2GjDDKXoVGFuvURtjDurWLor7zuvPwvQ9vLZgk9fhmADtyDlAx5gWhIa4M6Wgm4mjK7DV7/M2p8xfH6CPiMwTkYUiMsZvW6SIpDjlFcmhPbBPVUtrOCYAInK9s39KVlbWkV+NMc3UZcndGNU3nkf+t4ZNuwu8DscEwPfwnzvdVOD94HgYkASMAsYD00SkjbOtu6omA5cDj4tIr7ocWFWfV9VkVU2Oj4+vz5iNaVZEhEcuGkxEaAh3vbecMuuyavQyctx7hgPcTRzbgW5+nxOcMn/bgJmqWqKqG4F1+BIJqrrd+TcdmAMcA2QDbUQkrIZjGmPqWafYSP56wUBSNu/lpbkbvQ7H1EBV2bHvAF2CNHEsBpKcu6AigHHAzEp1PsLX2kBE4vB1XaWLSFsRaeFXPgJIVVUFZgOXOPtfDXzs4jUYYxy/PaYrZw7oyD9nrWHu+t1eh2OqsW9/CUWl5a49wwEuJg5nHOJWYBawGnhXVVeJyAMiUnGX1CwgW0RS8SWESaqaDfQHUkRkuVP+iKpWrG15D/AHEUnDN+bxolvXYIz5hYjwr0uH0Cu+FTe8nsKKbTleh2Sq8MvDf+61OMT3R3zTlpycrCkpKV6HYUyTsCu3kIuenk9hSRkzbjqJxLhor0Myfr5O3cV1r6Xw0S0jGNqtTe071EBEljhjzYfwenDcGBNkOraO5LUJwylX5aqXFpGVV+R1SMZPRm7Fw3/BOcZhjGmiesW34uVrh7Mrt4gHPk2tfQfTYDL2HSAsRGjfyp2H/8AShzHmMA3t1oabTu3FJ8t3MH+DDZY3Fhk5hXRsHenaw39gicMYcwRuGtWLbu1acv/HqygpK/c6HINvgsMuLk1uWMEShzHmsEWGh/KX8wayPjOfV+dv8jocQ8XDf+7diguWOIwxR+iM/h0Y3Teex75ax65cW6vcS6pKRk6hqwPjYInDGHOERIT7zx9ISZly30crKbUuK8/sKSimuLTc1Wc4wBKHMaYe9IiL5q6z+/Bl6i4mvJpCXqEtOeuFinU4rKvKGBMUrh/Zi4cvGsTctN1c+uwCtu874HVIzc4O52dug+PGmKAxfvhRvHrtcLbvPcDYqfPYkm0rBzaknYik1hQAABXCSURBVLkVLQ5LHMaYIHJyUhzv33wSeYUl/Peb9V6H06zs2FdIeKgQF+3ew39gicMY44I+HWO44vjufLRsu61X3oAycg7QKTaSEBcf/gNLHMYYl1w/siehIjzz3QavQ2k2MnIK6dza3YFxsMRhjHFJx9aRXHZcAjNStpGRYwPlDSE9q4Du7aNcP48lDmOMa24Y2YtyVZ7/Pt3rUJq87PwiducX0bdTjOvnssRhjHFNt3ZR/PaYrrz945aD06+rKgeKyzyOrOlZuzMPgH6dWrt+rrDaqxhjzOG7aVQv3l+6jatf+hGAzdkFFJaW8/I1xzGyT7zH0TUda5zEYS0OY0zQ6xnfimtHJHKgpIyOrVtwaXI3ureL4p73fybXnjCvN2t35tE+OoL4GHdvxQVrcRhjGsB95w3gvvMGHPy8bOs+Lnp6Hg99upp/XDLYw8iajjU7cxuktQHW4jDGeGBotzZcP7IX76RsZc7aTK/DCXrl5cq6XflNI3GIyBgRWSsiaSIyuZo6l4lIqoisEpG3nLKhIrLAKftZRH7nV/8VEdkoIsuc11A3r8EY4447zkgiqUMrpnywwrqsjtCWPfs5UFJGv2BPHCISCkwFzgEGAONFZEClOknAFGCEqg4E7nA27QeucsrGAI+LSBu/XSep6lDntcytazDGuCcyPJRHLx3CrtxCHvjE1i0/Er8MjLt/RxW42+IYDqSparqqFgPTgQsr1ZkITFXVvQCqmun8u05V1zvvdwCZgN1+YUwTM7RbG24Z3ZsZS7Yxc/kOr8MJWmt35iECfTq2apDzuZk4ugJb/T5vc8r89QH6iMg8EVkoImMqH0REhgMRgP+8BQ85XViPiUiVtxCIyPUikiIiKVlZWUd2JcYY19x+ehLHHtWGP32wwua1Okxrd+VyVLsooiIa5n4nrwfHw4AkYBQwHpjm3yUlIp2B14FrVbViWbEpQD/gOKAdcE9VB1bV51U1WVWT4+OtsWJMYxUeGsJ/xx0DwO3Tf6LEVhCsszU78+jbsWHGN8DdxLEd6Ob3OcEp87cNmKmqJaq6EViHL5EgIq2Bz4A/qerCih1UNUN9ioCX8XWJGWOCWLd2Ufy/iwbx05Z9/Pdrm4q9LgpLyti0u6DBBsbB3cSxGEgSkUQRiQDGATMr1fkIX2sDEYnD13WV7tT/EHhNVWf47+C0QhARAcYCK128BmNMAzl/SBcuS05g6pw0vk7d5XU4QSMtM59ybbiBcXAxcahqKXArMAtYDbyrqqtE5AERucCpNgvIFpFUYDa+u6WygcuAkcA1Vdx2+6aIrABWAHHA3926BmNMw/rbBUdzdJdYbp/+E6k7cr0OJyg05FQjFURVG+xkXklOTtaUlBSvwzDGBGBXbiEXPjWPEIGPbhlBh9buLoMa7B76LJVXF2wm9W9nExZav20BEVmiqsmVy70eHDfGmEN0bB3JC1cns3d/CRNfS7GZdGuxZmceSR1a1XvSqIklDmNMo3N011j+O24oP2/P4Y/vLaO8vOn3jByutTvzGrSbCixxGGMaqbMGduLec/rz+Yqd/OvLtV6H0yjtLSgmM6+I/g04MA42O64xphG77pRE0ncX8PScDfSIi+ay5G6179SMrN3lGxjv08AtDkscxphGS0R44MKBbN2zn3s/WEFC25ac1CvO67AajU27CwDoGRfdoOe1ripjTKMWHhrC1CuOpUdcNDe9sZTM3EKvQ2o0NmXvJzxU6NKmZYOe1xKHMabRi20ZzrSrkiksKeNvNpPuQZuzC+jWLorQEGnQ81riMMYEhcS4aG4/PYnPVmTwzWp7shx8LY4e7Ru2mwoscRhjgsjEU3rSp2Mr7vtoJQVFpV6H4ylVZXN2Ad3bRzX4uS1xGGOCRkRYCA9fNJiM3EL+/eU6r8PxVFZ+EfuLy6zFYYwxtRnWvS1XHH8Ur8zfyE9b9nodjmc2Z/vWLrEWhzHGBODuMf3o1DqSCa+msNaZ5K+5qbgV11ocxhgTgNaR4bw58QTCQ4XLpy1slsljy579hIYIXds27K24YInDGBOkEuOieXviCYQ5yWPdruaVPDZl7yehbUvCG3BywwqWOIwxQatnfCvenngCoSHCFS8sIju/yOuQGozvjqqG76YCSxzGmCDXM74Vr00YTs6BEqZ8sILmsMaQqrJxdwE9PBgYB0scxpgmoF+n1tx9dl++TN3Fe0u2eR2O6/btLyGvsNRaHMYYcyR+PyKRE3q2428zV7F1z36vw3HVpuyKO6qsxWGMMYctJET416VDCBHhj+8up6wJL/70yzMcTbDFISJjRGStiKSJyORq6lwmIqkiskpE3vIrv1pE1juvq/3Kh4nICueYT4hIw87uZYxptBLaRvHXCwby46Y9jHjkW256YwnPf7+BVTtyvA6tXm3KLkAEurVr+FtxwcX1OEQkFJgKnAlsAxaLyExVTfWrkwRMAUao6l4R6eCUtwPuB5IBBZY4++4FngEmAouAz4ExwBduXYcxJrhcdGxXFPhuXRbLtu7li5U7ARie2I7rT+nJaf06ENLAs8nWt83Z++kS25IWYaGenN/NhZyGA2mqmg4gItOBCwH/OZEnAlOdhICqZjrlZwNfqeoeZ9+vgDEiMgdoraoLnfLXgLFY4jDGOESES4YlcMmwBACy8or4eNl2Xpq7keteS6FXfDT3ntuf0/t39DjSw7cpu4Aecd6Mb4C7XVVdga1+n7c5Zf76AH1EZJ6ILBSRMbXs29V5X9MxARCR60UkRURSsrKyjuAyjDHBLD6mBded0pPv7h7Nf8cNRUSY8GoKt7y1lMy84FwUanP2fs/GN8D7wfEwIAkYBYwHpolIm/o4sKo+r6rJqpocHx9fH4c0xgSx8NAQLhzalc9vP4U/ntmHr1bt4ox/f8fHy7Z7HVqd5BwoYU9BsWd3VIG7iWM74L+yfIJT5m8bMFNVS1R1I7AOXyKpbt/tzvuajmmMMdWKCAvhttOT+OKOU+jTMYY731nGt2uCZ2GoLR7fUQXuJo7FQJKIJIpIBDAOmFmpzkf4WhuISBy+rqt0YBZwloi0FZG2wFnALFXNAHJF5ATnbqqrgI9dvAZjTBPVy3nifECX1tz21k+szsj1OqSA/PIMRxNMHKpaCtyKLwmsBt5V1VUi8oCIXOBUmwVki0gqMBuYpKrZzqD4g/iSz2LggYqBcuBm4AUgDdiADYwbYw5TVEQYL159HDGR4Ux4ZTGZuY1/zGOzkziOauddV5U0h3ldkpOTNSUlxeswjDGN1MrtOVz67AL6dGzFq78fTpuoCK9DqtZd7y3nh/VZLLr3DNfPJSJLVDW5crnXg+PGGOO5o7vG8t9xQ1m5I5dTH53DK/M2UlJW7nVYv5KZW8iXq3YyqGusp3FY4jDGGOCsgZ349LaTGdilNX/9JJVz/vsDC9OzvQ7rIFXlvo9XUlhazpRz+3saiyUOY4xx9O/cmjevO55pVyVTUlbOlS8u4n/Ok+eHY92uPB7+YjVrdh75wPvnK3Yya9Uu/nBmH3rFtzri4x0JSxzGGONHRDhzQEdm3noyg7rGcstbS/nop7rf9a+q3D3jZ577Lp0xj//A755bwBcrMg5r8sW9BcXcP3Mlg7rGct3JiXXev75Z4jDGmCrEtgzn9QnHM7xHO+58dxlv/7ilTvt/sXIny7bu48+/6c/kc/qxbe8BbnpzKTe+sYTi0rqNnzzwaSr79pfwz0sGE+bBUrGVeR+BMcY0UtEtwnj52uMY1SeeKR+sYPL7P5NXWFLrfsWl5fzzf2vo2zGGa0ckcuOpvfj+7tH8+Tf9+Sp1Fze9sYSi0rKAYliyeQ8f/rSdm0f3pn/n1kd6SfXCEocxxtQgMjyU565M5sZTe/FuylbGPP4Dc9fvRlXZtnc/s9dk8snyHYe0It7+cQubsvcz+Zx+hDoz8YaGCNed0pMHxx7NN2syueH1JRSW1J48nv0unTZR4dx4ak/XrrGu3Jwd1xhjmoSIsBAmn9OPswd25I/vLef/XlxEdEQoBcW/fPH36diKv48dRP/OMTzxzXpO7NmeUX1/PU/elSd0JyxEmPLBCm54fQkvXp1cbfdTWmY+X6/exW2jexMV0Xi+rhtPJMYY08gdc1RbPr/9FKZ9n87u/CKSOsbQt1MM2fnFPPhpKpc9t4C+HWPILihmyrn9qG6dufHDj0IV7v1wBY98sYY/nzegynov/JBORGgIV53Uw8WrqjtLHMYYUweR4aHcdnrSr8pH9onjiW/SeOGHdC4c2oXBCTVP9H358UexdmcuL8zdyKCEWC4ceugKEZl5hXywdDuXJCcQ16pFvV7DkbLEYYwx9SAqIozJ5/Tj9yf3ILZleED7/Pm8Aazemcc97/9Mr/hWHO33RPir8zdRUl7OxFMaz9hGBRscN8aYetQhJjLgJV3DQ0N4+opjaRsVwQ2vL2HBhmwOFJdRUFTKGwu3cNaAjiTGeTcLbnWsxWGMMR6Ka9WC564cxrjnFzJ+2kLCQoROsZHkHCjh+pG9vA6vSpY4jDHGY4MT2rBg8uks3bKXJZv3krJ5Dyf0bM+w7m29Dq1KljiMMaYRiI0KZ3S/Dozu18HrUGplYxzGGGPqxBKHMcaYOrHEYYwxpk4scRhjjKkTSxzGGGPqxNXEISJjRGStiKSJyOQqtl8jIlkissx5XeeUj/YrWyYihSIy1tn2iohs9Ns21M1rMMYYcyjXbscVkVBgKnAmsA1YLCIzVTW1UtV3VPVW/wJVnQ0MdY7TDkgDvvSrMklVZ7gVuzHGmOq52eIYDqSparqqFgPTgQsP4ziXAF+o6v56jc4YY8xhcfMBwK7AVr/P24Djq6h3sYiMBNYBd6rq1krbxwH/qVT2kIj8BfgGmKyqRZUPKiLXA9c7H/NFZC0QC+Q4ZbW9r/g3Dthdw3VWxf94gW6vXFbT58oxBlOs/mX1HW912wKJr7a4m3Os1W1vLr+zXsdaXXxVxepfVh/xdq+ylqq68sLXUnjB7/OVwFOV6rQHWjjvbwC+rbS9M5AFhFcqE6AF8CrwlzrE9Hyg7/3+TTmMa3++rtsrl9X0uYoYgyZWN+Otblsg8TX070EwxerG74HFGnisdflv73a8FS83u6q2A938Pic4ZQeparb+0lp4ARhW6RiXAR+qaonfPhnqUwS8jK9LLFCf1OG9f1ld1bZvVdsrl9X0uXKMwRRrIOesazy1bQskvureW6zVb28uv7Nex1q5zKv/vw4SJ8vUOxEJw9f9dDq+hLEYuFxVV/nV6ayqGc773wL3qOoJftsXAlPUN1h+yD7iW1rrMaBQVX91x1Y9XkeKqia7dfz6FEyxQnDFa7G6w2J1j5vxujbGoaqlInIrMAsIBV5S1VUi8gC+JtRM4HYRuQAoBfYA11TsLyI98LVYvqt06DdFJB5fd9Uy4Ea3rsHxvMvHr0/BFCsEV7wWqzssVve4Fq9rLQ5jjDFNkz05bowxpk4scRhjjKkTSxzGGGPqxBLHERCRU0TkWRF5QUTmex1PTUQkREQeEpEnReRqr+OpiYiMEpEfnJ/tKK/jqY2IRItIioic53UstRGR/s7PdYaI3OR1PDURkbEiMk1E3hGRs7yOpyYi0lNEXhSRRjkVkvM7+qrz87ziSI/XbBOHiLwkIpkisrJSeY0TM/pT1R9U9UbgU3wPIzbaWPFN95IAlOB7ir8xx6pAPhAZBLEC3AO8606Uh8RVH7+zq53f2cuAEY081o9UdSK+Oyd/18hjTVfVCW7FWJU6xn0RMMP5eV5wxCev65OFTeUFjASOBVb6lYUCG4CeQASwHBgADMKXHPxfHfz2exeIacyxApOBG5x9ZzTyWEOc/ToCbzbyWM/ENy3ONcB5wfA7i++L4wt8z1U16lid/f4NHBsksbr2/9YRxj0FGOrUeetIz+3mXFWNmqp+7zwr4u/gxIwAIjIduFBVHwaq7IYQkaOAHFXNa8yxisg2oNj5WNaYY/WzF9/UMq6op5/rKCAa3/+cB0Tkc1Utb6zxOseZCcwUkc+AtxprrM5Dvo/gm+R0qRtx1lesXqhL3Pha7gn4nn074p6mZps4qhHoxIz+JuCb+qSh1TXWD4AnReQU4Hs3A6tCnWIVkYuAs4E2wFPuhvYrdYpVVf8EvrVlgN1uJY0a1PVnOwpft0UL4HNXI/u1uv7O3gacAcSKSG9VfdbN4Cqp68+1PfAQcIyITHESjBeqi/sJ4CkR+Q1HNiUJYInjiKnq/V7HEAj1TUvfoH2wh0tVP8CX6IKGqr7idQyBUNU5wByPwwiIqj6B7wuv0VPVbNyfxeKwqWoBcG19Ha/ZDo5Xo9aJGRsRi9UdwRQrBFe8Fqv7GiRuSxyHWgwkiUiiiETgG/Sc6XFM1bFY3RFMsUJwxWuxuq9h4m6oOwAa2wt4G8jgl9tTJzjl5+Kb1XcD8Cev47RYLdZgjNdibdpx2ySHxhhj6sS6qowxxtSJJQ5jjDF1YonDGGNMnVjiMMYYUyeWOIwxxtSJJQ5jjDF1YonDNEsikt/A56uX9VrEt1ZJjogsE5E1IvKvAPYZKyID6uP8xoAlDmPqhYjUOO+bqp5Uj6f7QVWHAscA54lIbetqjMU3e68x9cIShzEOEeklIv8TkSXiW4Gwn1N+vogsEpGfRORrEenolP9VRF4XkXnA687nl0Rkjoiki8jtfsfOd/4d5Wyf4bQY3nSmD0dEznXKlojIEyLyaU3xquoBfNNkd3X2nygii0VkuYi8LyJRInISvvU3HnVaKb2qu05jAmWJw5hfPA/cpqrDgLuAp53yucAJqnoMMB2422+fAcAZqjre+dwP35Tww4H7RSS8ivMcA9zh7NsTGCEikcBzwDnO+eNrC1ZE2gJJ/DJN/geqepyqDgFW45uCYj6+uYomqepQVd1Qw3UaExCbVt0YQERaAScB7zkNAPhlEakE4B0R6YxvVbWNfrvOdP7yr/CZqhYBRSKSiW8Vw8rL3/6oqtuc8y4DeuBbKjddVSuO/TZwfTXhniIiy/EljcdVdadTfrSI/B3fOiatgFl1vE5jAmKJwxifEGCfM3ZQ2ZPAf1R1prMQ0l/9thVUqlvk976Mqv8fC6ROTX5Q1fNEJBFYKCLvquoy4BVgrKoudxaWGlXFvjVdpzEBsa4qYwBVzQU2isil4Fu2VESGOJtj+WVNg6tdCmEt0NNvKdDf1baD0zp5BLjHKYoBMpzusSv8quY522q7TmMCYonDNFdRIrLN7/UHfF+2E5xuoFX41moGXwvjPRFZAux2Ixinu+tm4H/OefKAnAB2fRYY6SSc+4BFwDxgjV+d6cAkZ3C/F9VfpzEBsWnVjWkkRKSVquY7d1lNBdar6mNex2VMZdbiMKbxmOgMlq/C1z32nMfxGFMla3EYY4ypE2txGGOMqRNLHMYYY+rEEocxxpg6scRhjDGmTixxGGOMqRNLHMYYY+rk/wNGYCc3t7gfsgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.511118</td>\n",
       "      <td>0.506876</td>\n",
       "      <td>0.788000</td>\n",
       "      <td>0.790099</td>\n",
       "      <td>0.827801</td>\n",
       "      <td>0.755682</td>\n",
       "      <td>00:12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>precision_score</th>\n",
       "      <th>recall_score</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.453230</td>\n",
       "      <td>0.479938</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.805068</td>\n",
       "      <td>0.829317</td>\n",
       "      <td>0.782197</td>\n",
       "      <td>00:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.432279</td>\n",
       "      <td>0.463301</td>\n",
       "      <td>0.821000</td>\n",
       "      <td>0.828380</td>\n",
       "      <td>0.838835</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.414865</td>\n",
       "      <td>0.439383</td>\n",
       "      <td>0.838000</td>\n",
       "      <td>0.846008</td>\n",
       "      <td>0.849237</td>\n",
       "      <td>0.842803</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.389433</td>\n",
       "      <td>0.439651</td>\n",
       "      <td>0.838000</td>\n",
       "      <td>0.845124</td>\n",
       "      <td>0.853282</td>\n",
       "      <td>0.837121</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.368743</td>\n",
       "      <td>0.424290</td>\n",
       "      <td>0.853000</td>\n",
       "      <td>0.859330</td>\n",
       "      <td>0.868472</td>\n",
       "      <td>0.850379</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.352984</td>\n",
       "      <td>0.427471</td>\n",
       "      <td>0.851000</td>\n",
       "      <td>0.855199</td>\n",
       "      <td>0.878244</td>\n",
       "      <td>0.833333</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.341775</td>\n",
       "      <td>0.422503</td>\n",
       "      <td>0.852000</td>\n",
       "      <td>0.856867</td>\n",
       "      <td>0.875494</td>\n",
       "      <td>0.839015</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.326303</td>\n",
       "      <td>0.421930</td>\n",
       "      <td>0.851000</td>\n",
       "      <td>0.857416</td>\n",
       "      <td>0.866538</td>\n",
       "      <td>0.848485</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.326604</td>\n",
       "      <td>0.420502</td>\n",
       "      <td>0.852000</td>\n",
       "      <td>0.857418</td>\n",
       "      <td>0.872549</td>\n",
       "      <td>0.842803</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.319492</td>\n",
       "      <td>0.419501</td>\n",
       "      <td>0.852000</td>\n",
       "      <td>0.857692</td>\n",
       "      <td>0.871094</td>\n",
       "      <td>0.844697</td>\n",
       "      <td>00:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fine_tune(10, 8e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.export('basic.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def infer_predictions(learn, df, cols):\n",
    "  tokenized_df = tokenize_df(df, text_cols=cols)\n",
    "  test_dl = learn.dls.test_dl(tokenized_df[0])\n",
    "  result = learn.get_preds(dl=test_dl, with_decoded=True)\n",
    "  \n",
    "  # Prediction, Confidence \n",
    "  _, y = learn.dls.valid.vocab\n",
    "  y_predicted = np.array(y[predicted]);\n",
    "  confidence = torch.max(result[0], axis=-1).values;\n",
    "\n",
    "  return y_predicted, confidence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  return array(a, dtype, copy=False, order=order)\n"
     ]
    }
   ],
   "source": [
    "tokenized_df = tokenize_df(test_df, text_cols=['Text', 'HashTag'])\n",
    "test_dl = learn.dls.test_dl(tokenized_df[0], with_labels=True)\n",
    "result = learn.get_preds(dl=test_dl, with_decoded=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2000, 2]), torch.Size([2000]), torch.Size([2000]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result[0].shape, result[1].shape, result[2].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.9495, 0.8612, 0.9765,  ..., 0.8892, 0.5078, 0.9341])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confidence = torch.max(result[0], axis=-1).values;confidence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['UNINFORMATIVE', 'UNINFORMATIVE', 'UNINFORMATIVE', ...,\n",
       "       'INFORMATIVE', 'UNINFORMATIVE', 'INFORMATIVE'], dtype='<U13')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "  _, y = learn.dls.valid.vocab\n",
    "  y_predicted = np.array(y[predicted]);y_predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['confidence'] = confidence\n",
    "test_df['predicted'] = y_predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, torch.Size([2000, 2]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(result), result[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['accuracy', 'f1_score', 'precision_score', 'recall_score']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[each.name for each in learn.metrics]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(#5) [0.4195009768009186,0.8519999980926514,0.8576923076923078,0.87109375,0.8446969696969697]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.validate(ds_idx=1) # Validation Set got 85% accuracy and 0.857 f1 score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(#5) [0.49813932180404663,0.796500027179718,0.8229665071770335,0.7610619469026548,0.8958333333333334]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.validate(dl=test_dl) # Room for improvement for test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0505, 0.9495],\n",
       "        [0.1388, 0.8612],\n",
       "        [0.0235, 0.9765],\n",
       "        ...,\n",
       "        [0.8892, 0.1108],\n",
       "        [0.4922, 0.5078],\n",
       "        [0.9341, 0.0659]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['UNINFORMATIVE', 'INFORMATIVE', 'UNINFORMATIVE', ...,\n",
       "       'INFORMATIVE', 'INFORMATIVE', 'INFORMATIVE'], dtype=object)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.Label.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, ..., 0, 0, 0])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Numericalize(vocab=['INFORMATIVE', 'UNINFORMATIVE'])(test_df.Label).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cleanlab.pruning import get_noise_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordered_label_errors = get_noise_indices(s=Numericalize(vocab=['INFORMATIVE', 'UNINFORMATIVE'])(test_df.Label).numpy(), \n",
    "                                         psx=result[0].numpy(), \n",
    "                                         sorted_index_method='normalized_margin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "194"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ordered_label_errors) # We have identified around 194 problematic label errors in the test data itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UNINFORMATIVE    1056\n",
       "INFORMATIVE       944\n",
       "Name: Label, dtype: int64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.Label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1358, 1177,  398, 1694, 1501,  252, 1129,  852, 1353, 1174,   35,\n",
       "        171, 1792,  242,  656, 1318, 1506, 1951, 1688, 1216,  522,  559,\n",
       "       1978, 1937, 1064,  667,  376,  923,  440, 1108,   39, 1800, 1227,\n",
       "        617,  156,  738,  290,  328,  636,  905,  768, 1558, 1355, 1971,\n",
       "        130, 1392,  997,   27, 1235, 1657,   42,  570, 1677, 1156,  218,\n",
       "       1740,  528,  211, 1825, 1898,   82, 1199, 1919, 1641, 1436, 1389,\n",
       "        138, 1538,  225,    1, 1608, 1395, 1830, 1459, 1308,  129,  573,\n",
       "       1224, 1217,  613, 1495, 1257, 1845, 1682,  316,  438, 1153,  904,\n",
       "       1915,  827,  206,  117,  278,  870,  677, 1948, 1875, 1144, 1124,\n",
       "        924,  652,  614, 1612,  878, 1077, 1255, 1734,  987, 1489, 1640,\n",
       "       1851,   29, 1706,  775, 1375,  224,  286,  912,  421,  577, 1562,\n",
       "       1904, 1098,  750, 1666,  773,  992,  392, 1135, 1559, 1555,  448,\n",
       "       1471,  759,  977, 1137, 1415,  423,  196,  975, 1907, 1193, 1306,\n",
       "        603,  329,  526,  524,  365,  877,   62,  102, 1920,   19,  956,\n",
       "         33,  385,  263, 1946,   46, 1105, 1016, 1332, 1678, 1909, 1972,\n",
       "       1042, 1321, 1620, 1402, 1542, 1204,   43,  550,  443,  909,  179,\n",
       "        470, 1604, 1759, 1649, 1141, 1639, 1870,  785, 1569, 1777, 1231,\n",
       "       1084,  549,  858,  611,  285,  732, 1744])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ordered_label_errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['confidenceRange'] = pd.cut(test_df['confidence'], bins=[0.0, 0.6, 0.9, 1.0], labels=['<60', '60<90', '90+'],include_lowest=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60<90    133\n",
       "90+       51\n",
       "<60       10\n",
       "Name: confidenceRange, dtype: int64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Out of 194 samples, we can able to confirm 51 of them are with greater than 90% confidence being incorrect labeling\n",
    "test_df.iloc[ordered_label_errors]['confidenceRange'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing the models in GDrive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/content/gdrive/My Drive/Colab Notebooks/covid/models/basic.pkl'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shutil.copy('./models/basic.pkl', drive_data + '/models')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/content/gdrive/My Drive/Colab Notebooks/covid/models/fine_tuned_enc.pth'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shutil.copy('./models/fine_tuned_enc.pth', drive_data + '/models')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}