PageSpeed Insights Mass Carwler (aka psinsightsmasscrawler)

Psinsightsmasscrawler is a tool to make - mass - PageSpeed tests easier and obtain a clear report.

Just add a domain, crawl website's URLs (using sitemap.whatever, do not worry it's automagical), create a batch (a list of URLs to analyze) and run the batch to get a clean view of a website performance.

This tool store the report as a JSON downloadable file that you can analyse using Lighthouse Report Viewer.

Have a look at this YouTube video https://www.youtube.com/watch?v=AVuUznzflug for a complete demo.

Requirements

1. Pagespeed API keys (2 keys for async workers), see Get Started with the PageSpeed Insights API
2. Clone this repo:

$ git clone https://github.com/TeddyBear06/ps-insights-mass-crawler.git

3. Docker (tested with 4.30, latest actually)

Quickstart

[1/4] Run the stack:

$ docker compose -p psinsightsmasscrawler up -d

[2/4] Open the app container shell:

$ docker exec -it psinsightsmasscrawler-app-1 sh

[3/4] Create a django admin super user (follow CLI instructions):

$ python manage.py createsuperuser

[4/4] Go to http://127.0.0.1:8000/admin/

Useful commands

Remove recents actions

Open a django shell:

$ python manage.py shell

Then:

>>> from django.contrib.admin.models import LogEntry
>>> LogEntry.objects.all().delete()

Dependencies

Thanks to all contributors of thoose wonderful projects:

• Django
• ultimate-sitemap-parser
• redis
• Celery