scraper_vinfolio

Web Scraper that use Selenium Webdriver

Clone the repository

git clone https://github.com/mosiahr/scraper_vinfolio.git
cd scraper_vinfolio

Create virtual environment and activate it

virtualenv -p python3 venv
source venv/bin/activate
pip install -r requirements.txt

Run

python core/scraper

Task 1. Find all Items

Use:

@run_time
def main():
    scraper = Scraper(BASE_URL, profile=True)
    scraper.parsing()

Task 2. Find Items Without Pictures

Use class ScraperWithoutImg:

@run_time
def main():
    scraper = ScraperWithoutImg(BASE_URL)
    scraper.parsing()

Other

To disable the loading of CSS, images, Flash, WZ, set the class Scraper attribute profile=True
Disable everything but leave javascript enabled use js_enabled: profile=True, js_enabled=True