class_15_20200202_Libraries.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "- Making Profile at UpWork\n",
    "  1. Sending Proposal\n",
    "     a. Decompression Zone\n",
    "     b. Avoid \"Hybrid Proposal\" - sending same proposal to everyone\n",
    "     c. Offer a helpusl suggestion right in your cover letter\n",
    "     d. Don't burry the lead. write short story but more practical\n",
    "     e. Show an example of similar work (if you're brand-new)\n",
    "     f. Find common ground - find similarities fo build trust\n",
    "     g. Become a screening question \"smuggler\" if client asks additional questions then, it sould be answered first even     before the cover letter.\n",
    "     h. End your cover letter with a \"Call To Action\" - to offer more conversation before finalizing project\n",
    "     j. Don't sell yourself short - You can even demand more amount even employer's allocated budget\n",
    "     k. Address employer with his/her name\n",
    "     l. Don't be a widow - don't beg projects\n",
    "     m. One word = 1 Year - be specific and write important point in very start\n",
    "     n. Additional Tip: Threat your Client - Tell employer that I am expert if you don't give me project you will have such, such loss and I am the only person who can do it well.\n",
    "   2. Play strategally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (1)  PyAutoGUI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation Commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Installation will wil done at command prompt\n",
    "# pyautogui is for desktop applications"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# winodws\n",
    "$ pip install pyautogui\n",
    "\n",
    "# Mac\n",
    "$ pip3 install pyobjc-core\n",
    "$ pip3 install pyobjc\n",
    "$ pip3 install pyautogui\n",
    "\n",
    "# Linux\n",
    "$ pip3 install python3-xlimb\n",
    "$ pip3 install pyautogui"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyAutoGUI Commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyautogui as pag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'OK, Thanks'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to create alert box \n",
    "# press ok to get \"ok\" message\n",
    "\n",
    "pag.alert(\"Please click 'Ok'\",\"alert\",button=\"OK\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-36-2b3adfe506ed>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-36-2b3adfe506ed>\"\u001b[1;36m, line \u001b[1;32m2\u001b[0m\n\u001b[1;33m    if btn = \"Ok\":\u001b[0m\n\u001b[1;37m           ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "btn = pag.confirm(text=\"\",title=\"\",button=['Ok', 'Cancel'])\n",
    "if btn = \"Ok\":\n",
    "    print(\"You selected\")\n",
    "    print(\"Files deleted\")\n",
    "else:\n",
    "    print(\"File are not deleted\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your password is incorrect\n"
     ]
    }
   ],
   "source": [
    "btn=pag.password(text=\"Enter password\", title=\"Password Box\")\n",
    "if pag.password == 'abc':\n",
    "    print(\"You're logged in\")\n",
    "else:\n",
    "    print(\"Your password is incorrect\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=360, y=193)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pag.position()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cheat sheet of pyautogui is avalable at this link:\n",
    "\n",
    "# pyautogui.readthedocs.io  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.moveTo(1157, 18, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  pag.position(6000,6000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.click(1157, 18)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.moveTo(750, 17, 4)\n",
    "pag.click()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=385, y=488)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pag.position()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.moveTo(1167, 16)\n",
    "pag.click()\n",
    "\n",
    "paag.moveTo(814, 421, 4)\n",
    "pag.click()\n",
    "\n",
    "pag.write(\"pyautogui documentation\\n\", interval = 0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.hotkey(\"winleft\",\"up\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.hotkey(\"winleft\",\"left\")\n",
    "pag.PAUSE=2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Size(width=1366, height=768)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  to see screen size\n",
    "pag.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=357, y=749)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to check position of cursor\n",
    "pag.position()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "pag.moveTo(357, 749, 2)\n",
    "pag.click()\n",
    "pag.write(\"calc\\n\", interval=0.15)\n",
    "pag.write(\"5+3\\n\", interval=0.10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (2)  Beautifulsoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This library is used for web-scraping (to collect data from website)\n",
    "- Web scraping is very important for every AI Engr, ML Engr & Data Scientist\n",
    "- UCI is a machine learning repository, googledataset is also google dataset repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'error', 'parse', 'request', 'response']\n"
     ]
    }
   ],
   "source": [
    "print(dir(urllib))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on package urllib:\n",
      "\n",
      "NAME\n",
      "    urllib\n",
      "\n",
      "PACKAGE CONTENTS\n",
      "    error\n",
      "    parse\n",
      "    request\n",
      "    response\n",
      "    robotparser\n",
      "\n",
      "FILE\n",
      "    c:\\programdata\\anaconda3\\lib\\urllib\\__init__.py\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(urllib)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "url = \"http://www.propakistani.pk/robots.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install beautifulsoup\n",
    "# pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/transitional.dtd\">\n",
      "<html>\n",
      "<head>\n",
      "<meta content=\"text/html; charset=utf-8\" http-equiv=\"Content-Type\"/>\n",
      "<title>Beautiful Soup: We called him Tortoise because he taught us.</title>\n",
      "<link href=\"mailto:leonardr@segfault.org\" rev=\"made\"/>\n",
      "<link href=\"/nb/themes/Default/nb.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "<meta content=\"Beautiful Soup: a library designed for screen-scraping HTML and XML.\" name=\"Description\"/>\n",
      "<meta content=\"Markov Approximation 1.4 (module: leonardr)\" name=\"generator\"/>\n",
      "<meta content=\"Leonard Richardson\" name=\"author\"/>\n",
      "</head>\n",
      "<body alink=\"red\" bgcolor=\"white\" link=\"blue\" text=\"black\" vlink=\"660066\">\n",
      "<style>\n",
      "#tidelift { }\n",
      "\n",
      "#tidelift a {\n",
      " border: 1px solid #666666;\n",
      " margin-left: auto;\n",
      " padding: 10px;\n",
      " text-decoration: none;\n",
      "}\n",
      "\n",
      "#tidelift .cta {\n",
      " background: url(\"tidelift.svg\") no-repeat;\n",
      " padding-left: 30px;\n",
      "}\n",
      "</style>\n",
      "<img align=\"right\" src=\"10.1.jpg\" width=\"250\"/><br/>\n",
      "<p>[ <a href=\"#Download\">Download</a> | <a href=\"bs4/doc/\">Documentation</a> | <a href=\"#HallOfFame\">Hall of Fame</a> | <a href=\"enterprise.html\">For enterprise</a> | <a href=\"https://code.launchpad.net/beautifulsoup\">Source</a> | <a href=\"https://bazaar.launchpad.net/%7Eleonardr/beautifulsoup/bs4/view/head:/CHANGELOG\">Changelog</a> | <a href=\"https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup\">Discussion group</a>  | <a href=\"zine/\">Zine</a> ]</p>\n",
      "<div align=\"center\">\n",
      "<a href=\"bs4/download/\"><h1>Beautiful Soup</h1></a>\n",
      "</div>\n",
      "<p>You didn't write that awful page. You're just trying to get some\n",
      "data out of it. Beautiful Soup is here to help. Since 2004, it's been\n",
      "saving programmers hours or days of work on quick-turnaround\n",
      "screen scraping projects.</p>\n",
      "<p>Beautiful Soup is a Python library designed for quick turnaround\n",
      "projects like screen-scraping. Three features make it powerful:\n",
      "\n",
      "</p><ol>\n",
      "<li>Beautiful Soup provides a few simple methods and Pythonic idioms\n",
      "for navigating, searching, and modifying a parse tree: a toolkit for\n",
      "dissecting a document and extracting what you need. It doesn't take\n",
      "much code to write an application\n",
      "\n",
      "</li><li>Beautiful Soup automatically converts incoming documents to\n",
      "Unicode and outgoing documents to UTF-8. You don't have to think\n",
      "about encodings, unless the document doesn't specify an encoding and\n",
      "Beautiful Soup can't detect one. Then you just have to specify the\n",
      "original encoding.\n",
      "\n",
      "</li><li>Beautiful Soup sits on top of popular Python parsers like <a href=\"http://lxml.de/\">lxml</a> and <a href=\"http://code.google.com/p/html5lib/\">html5lib</a>, allowing you\n",
      "to try out different parsing strategies or trade speed for\n",
      "flexibility.\n",
      "\n",
      "</li></ol>\n",
      "<p>Beautiful Soup parses anything you give it, and does the tree\n",
      "traversal stuff for you. You can tell it \"Find all the links\", or\n",
      "\"Find all the links of class <tt>externalLink</tt>\", or \"Find all the\n",
      "links whose urls match \"foo.com\", or \"Find the table heading that's\n",
      "got bold text, then give me that text.\"\n",
      "\n",
      "</p><p>Valuable data that was once locked up in poorly-designed websites\n",
      "is now within your reach. Projects that would have taken hours take\n",
      "only minutes with Beautiful Soup.\n",
      "\n",
      "</p><p>Interested? <a href=\"bs4/doc/\">Read more.</a>\n",
      "</p><h3>Getting and giving support</h3>\n",
      "<div align=\"center\" id=\"tidelift\">\n",
      "<a href=\"https://tidelift.com/subscription/pkg/pypi-beautifulsoup4?utm_source=pypi-beautifulsoup4&amp;utm_medium=referral&amp;utm_campaign=enterprise\" target=\"_blank\">\n",
      "<span class=\"cta\">\n",
      "  Beautiful Soup for enterprise available via Tidelift\n",
      " </span>\n",
      "</a>\n",
      "</div>\n",
      "<p>If you have questions, send them to <a href=\"https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup\">the discussion\n",
      "group</a>. If you find a bug, <a href=\"https://bugs.launchpad.net/beautifulsoup/\">file it on Launchpad</a>. If it's a security vulnerability, report it confidentially through <a href=\"https://tidelift.com/security\">Tidelift</a>.</p>\n",
      "<p>If you use Beautiful Soup as part of your work, please consider a <a href=\"https://tidelift.com/subscription/pkg/pypi-beautifulsoup4?utm_source=pypi-beautifulsoup4&amp;utm_medium=referral&amp;utm_campaign=website\">Tidelift subscription</a>. This will support many of the free software projects your organization depends on, not just Beautiful Soup.\n",
      "\n",
      "\n",
      "</p><p>If Beautiful Soup is useful to you on a personal level, you might like to read <a href=\"zine/\"><i>Tool Safety</i></a>, a short zine I wrote about what I learned about software development from working on Beautiful Soup. Thanks!</p>\n",
      "<a name=\"Download\"><h2>Download Beautiful Soup</h2></a>\n",
      "<p>The current release is <a href=\"bs4/download/\">Beautiful Soup\n",
      "4.8.2</a> (December 24, 2019). You can install Beautiful Soup 4 with\n",
      "<code>pip install beautifulsoup4</code>.\n",
      "\n",
      "</p><p>In Debian and Ubuntu, Beautiful Soup is available as the\n",
      "<code>python-bs4</code> package (for Python 2) or the\n",
      "<code>python3-bs4</code> package (for Python 3). In Fedora it's\n",
      "available as the <code>python-beautifulsoup4</code> package.\n",
      "\n",
      "</p><p>Beautiful Soup is licensed under the MIT license, so you can also\n",
      "download the tarball, drop the <code>bs4/</code> directory into almost\n",
      "any Python application (or into your library path) and start using it\n",
      "immediately. (If you want to do this under Python 3, you will need to\n",
      "manually convert the code using <code>2to3</code>.)\n",
      "\n",
      "</p><p>Beautiful Soup 4 works on both Python 2 (2.7+) and Python\n",
      "3. Support for Python 2 will be discontinued on or after December 31,\n",
      "2020—one year after the Python 2 sunsetting date.\n",
      "\n",
      "</p><h3>Beautiful Soup 3</h3>\n",
      "<p>Beautiful Soup 3 was the official release line of Beautiful Soup\n",
      "from May 2006 to March 2012. It does not support Python 3 and it will\n",
      "be discontinued on or after December 31, 2020—one year after the\n",
      "Python 2 sunsetting date. If you have any active projects using\n",
      "Beautiful Soup 3, you should migrate to Beautiful Soup 4 as part of\n",
      "your Python 3 conversion.\n",
      "\n",
      "</p><p><a href=\"http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html\">Here's\n",
      "the Beautiful Soup 3 documentation.</a>\n",
      "</p><p>The current and hopefully final release of Beautiful Soup 3 is <a href=\"download/3.x/BeautifulSoup-3.2.2.tar.gz\">3.2.2</a> (October 5,\n",
      "2019). It's the <code>BeautifulSoup</code> package on pip. It's also\n",
      "available as <code>python-beautifulsoup</code> in Debian and Ubuntu,\n",
      "and as <code>python-BeautifulSoup</code> in Fedora.\n",
      "\n",
      "</p><p>Once Beautiful Soup 3 is discontinued, these package names will be available for use by a more recent version of Beautiful Soup.\n",
      "\n",
      "</p><p>Beautiful Soup 3, like Beautiful Soup 4, is <a href=\"https://tidelift.com/subscription/pkg/pypi-beautifulsoup?utm_source=pypi-beautifulsoup&amp;utm_medium=referral&amp;utm_campaign=website\">supported through Tidelift</a>.</p>\n",
      "<a name=\"HallOfFame\"><h2>Hall of Fame</h2></a>\n",
      "<p>Over the years, Beautiful Soup has been used in hundreds of\n",
      "different projects. There's no way I can list them all, but I want to\n",
      "highlight a few high-profile projects. Beautiful Soup isn't what makes\n",
      "these projects interesting, but it did make their completion easier:\n",
      "\n",
      "</p><ul>\n",
      "<li><a href=\"http://www.nytimes.com/2007/10/25/arts/design/25vide.html\">\"Movable\n",
      " Type\"</a>, a work of digital art on display in the lobby of the New\n",
      " York Times building, uses Beautiful Soup to scrape news feeds.\n",
      "\n",
      "</li><li>Reddit uses Beautiful Soup to <a href=\"https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py\">parse\n",
      "a page that's been linked to and find a representative image</a>.\n",
      "\n",
      "</li><li>Alexander Harrowell uses Beautiful Soup to <a href=\"http://www.harrowell.org.uk/viktormap.html\">track the business\n",
      " activities</a> of an arms merchant.\n",
      "\n",
      "</li><li>The developers of Python itself used Beautiful Soup to <a href=\"http://svn.python.org/view/tracker/importer/\">migrate the Python\n",
      "bug tracker from Sourceforge to Roundup</a>.\n",
      "\n",
      "</li><li>The <a href=\"http://www2.ljworld.com/\">Lawrence Journal-World</a>\n",
      "uses Beautiful Soup to <a href=\"http://www.b-list.org/weblog/2010/nov/02/news-done-broke/\">gather\n",
      "statewide election results</a>.\n",
      "\n",
      "</li><li>The <a href=\"http://esrl.noaa.gov/gsd/fab/\">NOAA's Forecast\n",
      "Applications Branch</a> uses Beautiful Soup in <a href=\"http://laps.noaa.gov/topograbber/\">TopoGrabber</a>, a script for\n",
      "downloading \"high resolution USGS datasets.\"\n",
      "\n",
      "</li></ul>\n",
      "<p>If you've used Beautiful Soup in a project you'd like me to know\n",
      "about, please do send email to me or <a href=\"http://groups.google.com/group/beautifulsoup/\">the discussion\n",
      "group</a>.\n",
      "\n",
      "</p><h2>Development</h2>\n",
      "<p>Development happens at <a href=\"https://launchpad.net/beautifulsoup\">Launchpad</a>. You can <a href=\"https://code.launchpad.net/beautifulsoup/\">get the source\n",
      "code</a> or <a href=\"https://bugs.launchpad.net/beautifulsoup/\">file\n",
      "bugs</a>.</p><hr/><table><tr><td valign=\"top\">\n",
      "<p>This document (<a href=\"/source/software/BeautifulSoup/index.bhtml\">source</a>) is part of Crummy, the webspace of <a href=\"/self/\">Leonard Richardson</a> (<a href=\"/self/contact.html\">contact information</a>). It was last modified on Friday, January 31 2020, 13:44:05 Nowhere Standard Time and last built on Sunday, February 02 2020, 06:00:01 Nowhere Standard Time.</p><p></p><table class=\"licenseText\"><tr><td><a href=\"http://creativecommons.org/licenses/by-sa/2.0/\"><img border=\"0\" src=\"/nb//resources/img/somerights20.jpg\"/></a></td><td valign=\"top\">Crummy is © 1996-2020 Leonard Richardson. Unless otherwise noted, all text licensed under a <a href=\"http://creativecommons.org/licenses/by-sa/2.0/\">Creative Commons License</a>.</td></tr></table><!--<rdf:RDF xmlns=\"http://web.resource.org/cc/\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"><Work rdf:about=\"http://www.crummy.com/\"><dc:title>Crummy: The Site</dc:title><dc:rights><Agent><dc:title>Crummy: the Site</dc:title></Agent></dc:rights><dc:format>text/html</dc:format><license rdf:resource=http://creativecommons.org/licenses/by-sa/2.0//></Work><License rdf:about=\"http://creativecommons.org/licenses/by-sa/2.0/\"></License></rdf:RDF>--></td><td valign=\"top\"><p><b>Document tree:</b>\n",
      "</p><dl><dd><a href=\"http://www.crummy.com/\">http://www.crummy.com/</a><dl><dd><a href=\"http://www.crummy.com/software/\">software/</a><dl><dd><a href=\"http://www.crummy.com/software/BeautifulSoup/\">BeautifulSoup/</a></dd></dl>\n",
      "</dd></dl>\n",
      "</dd></dl>\n",
      "\n",
      "\n",
      "Site Search:\n",
      "\n",
      "<form action=\"/search/\" method=\"get\">\n",
      "<input maxlength=\"255\" name=\"q\" type=\"text\" value=\"\"/>\n",
      "</form>\n",
      "</td>\n",
      "</tr>\n",
      "</table>\n",
      "</body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "url = \"https://www.crummy.com/software/BeautifulSoup/\"\n",
    "r = requests.get(url)\n",
    "html_doc = r.text\n",
    "soup = BeautifulSoup(html_doc)   # to create beautifulsoup object\n",
    "print(soup)\n",
    "# soup.prettify()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>Beautiful Soup: We called him Tortoise because he taught us.</title>\n",
      "<class 'bs4.element.Tag'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Beautiful Soup: We called him Tortoise because he taught us.'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(soup.title)\n",
    "print(type(soup.title))\n",
    "soup.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Beautiful Soup: We called him Tortoise because he taught us.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "#tidelift { }\n",
      "\n",
      "#tidelift a {\n",
      " border: 1px solid #666666;\n",
      " margin-left: auto;\n",
      " padding: 10px;\n",
      " text-decoration: none;\n",
      "}\n",
      "\n",
      "#tidelift .cta {\n",
      " background: url(\"tidelift.svg\") no-repeat;\n",
      " padding-left: 30px;\n",
      "}\n",
      "\n",
      "\n",
      "[ Download | Documentation | Hall of Fame | For enterprise | Source | Changelog | Discussion group  | Zine ]\n",
      "\n",
      "Beautiful Soup\n",
      "\n",
      "You didn't write that awful page. You're just trying to get some\n",
      "data out of it. Beautiful Soup is here to help. Since 2004, it's been\n",
      "saving programmers hours or days of work on quick-turnaround\n",
      "screen scraping projects.\n",
      "Beautiful Soup is a Python library designed for quick turnaround\n",
      "projects like screen-scraping. Three features make it powerful:\n",
      "\n",
      "\n",
      "Beautiful Soup provides a few simple methods and Pythonic idioms\n",
      "for navigating, searching, and modifying a parse tree: a toolkit for\n",
      "dissecting a document and extracting what you need. It doesn't take\n",
      "much code to write an application\n",
      "\n",
      "Beautiful Soup automatically converts incoming documents to\n",
      "Unicode and outgoing documents to UTF-8. You don't have to think\n",
      "about encodings, unless the document doesn't specify an encoding and\n",
      "Beautiful Soup can't detect one. Then you just have to specify the\n",
      "original encoding.\n",
      "\n",
      "Beautiful Soup sits on top of popular Python parsers like lxml and html5lib, allowing you\n",
      "to try out different parsing strategies or trade speed for\n",
      "flexibility.\n",
      "\n",
      "\n",
      "Beautiful Soup parses anything you give it, and does the tree\n",
      "traversal stuff for you. You can tell it \"Find all the links\", or\n",
      "\"Find all the links of class externalLink\", or \"Find all the\n",
      "links whose urls match \"foo.com\", or \"Find the table heading that's\n",
      "got bold text, then give me that text.\"\n",
      "\n",
      "Valuable data that was once locked up in poorly-designed websites\n",
      "is now within your reach. Projects that would have taken hours take\n",
      "only minutes with Beautiful Soup.\n",
      "\n",
      "Interested? Read more.\n",
      "Getting and giving support\n",
      "\n",
      "\n",
      "\n",
      "  Beautiful Soup for enterprise available via Tidelift\n",
      " \n",
      "\n",
      "\n",
      "If you have questions, send them to the discussion\n",
      "group. If you find a bug, file it on Launchpad. If it's a security vulnerability, report it confidentially through Tidelift.\n",
      "If you use Beautiful Soup as part of your work, please consider a Tidelift subscription. This will support many of the free software projects your organization depends on, not just Beautiful Soup.\n",
      "\n",
      "\n",
      "If Beautiful Soup is useful to you on a personal level, you might like to read Tool Safety, a short zine I wrote about what I learned about software development from working on Beautiful Soup. Thanks!\n",
      "Download Beautiful Soup\n",
      "The current release is Beautiful Soup\n",
      "4.8.2 (December 24, 2019). You can install Beautiful Soup 4 with\n",
      "pip install beautifulsoup4.\n",
      "\n",
      "In Debian and Ubuntu, Beautiful Soup is available as the\n",
      "python-bs4 package (for Python 2) or the\n",
      "python3-bs4 package (for Python 3). In Fedora it's\n",
      "available as the python-beautifulsoup4 package.\n",
      "\n",
      "Beautiful Soup is licensed under the MIT license, so you can also\n",
      "download the tarball, drop the bs4/ directory into almost\n",
      "any Python application (or into your library path) and start using it\n",
      "immediately. (If you want to do this under Python 3, you will need to\n",
      "manually convert the code using 2to3.)\n",
      "\n",
      "Beautiful Soup 4 works on both Python 2 (2.7+) and Python\n",
      "3. Support for Python 2 will be discontinued on or after December 31,\n",
      "2020—one year after the Python 2 sunsetting date.\n",
      "\n",
      "Beautiful Soup 3\n",
      "Beautiful Soup 3 was the official release line of Beautiful Soup\n",
      "from May 2006 to March 2012. It does not support Python 3 and it will\n",
      "be discontinued on or after December 31, 2020—one year after the\n",
      "Python 2 sunsetting date. If you have any active projects using\n",
      "Beautiful Soup 3, you should migrate to Beautiful Soup 4 as part of\n",
      "your Python 3 conversion.\n",
      "\n",
      "Here's\n",
      "the Beautiful Soup 3 documentation.\n",
      "The current and hopefully final release of Beautiful Soup 3 is 3.2.2 (October 5,\n",
      "2019). It's the BeautifulSoup package on pip. It's also\n",
      "available as python-beautifulsoup in Debian and Ubuntu,\n",
      "and as python-BeautifulSoup in Fedora.\n",
      "\n",
      "Once Beautiful Soup 3 is discontinued, these package names will be available for use by a more recent version of Beautiful Soup.\n",
      "\n",
      "Beautiful Soup 3, like Beautiful Soup 4, is supported through Tidelift.\n",
      "Hall of Fame\n",
      "Over the years, Beautiful Soup has been used in hundreds of\n",
      "different projects. There's no way I can list them all, but I want to\n",
      "highlight a few high-profile projects. Beautiful Soup isn't what makes\n",
      "these projects interesting, but it did make their completion easier:\n",
      "\n",
      "\n",
      "\"Movable\n",
      " Type\", a work of digital art on display in the lobby of the New\n",
      " York Times building, uses Beautiful Soup to scrape news feeds.\n",
      "\n",
      "Reddit uses Beautiful Soup to parse\n",
      "a page that's been linked to and find a representative image.\n",
      "\n",
      "Alexander Harrowell uses Beautiful Soup to track the business\n",
      " activities of an arms merchant.\n",
      "\n",
      "The developers of Python itself used Beautiful Soup to migrate the Python\n",
      "bug tracker from Sourceforge to Roundup.\n",
      "\n",
      "The Lawrence Journal-World\n",
      "uses Beautiful Soup to gather\n",
      "statewide election results.\n",
      "\n",
      "The NOAA's Forecast\n",
      "Applications Branch uses Beautiful Soup in TopoGrabber, a script for\n",
      "downloading \"high resolution USGS datasets.\"\n",
      "\n",
      "\n",
      "If you've used Beautiful Soup in a project you'd like me to know\n",
      "about, please do send email to me or the discussion\n",
      "group.\n",
      "\n",
      "Development\n",
      "Development happens at Launchpad. You can get the source\n",
      "code or file\n",
      "bugs.\n",
      "This document (source) is part of Crummy, the webspace of Leonard Richardson (contact information). It was last modified on Friday, January 31 2020, 13:44:05 Nowhere Standard Time and last built on Sunday, February 02 2020, 06:00:01 Nowhere Standard Time.Crummy is © 1996-2020 Leonard Richardson. Unless otherwise noted, all text licensed under a Creative Commons License.Document tree:\n",
      "http://www.crummy.com/software/BeautifulSoup/\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Site Search:\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(soup.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (3) Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This library is used for testing websites\n",
    "- goto selenium website and install webdriver for chrome\n",
    "- unzip webdriver file at 'anaconda3' installation folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "cannot import name 'keys' from 'selenium.webdriver.common.keys' (C:\\ProgramData\\Anaconda3\\lib\\site-packages\\selenium\\webdriver\\common\\keys.py)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-e8c95a96a862>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mselenium\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mwebdriver\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mselenium\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwebdriver\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcommon\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkeys\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mkeys\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m: cannot import name 'keys' from 'selenium.webdriver.common.keys' (C:\\ProgramData\\Anaconda3\\lib\\site-packages\\selenium\\webdriver\\common\\keys.py)"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (4) Pyqt5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This library is used to make praphical buttons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import site"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# designer.exe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}