Skip to content

Data visualizer + AQI calculator for CSV-files provided by SaveEcoBot

License

Notifications You must be signed in to change notification settings

MaxymVlasov/eco-data-visualizer

Repository files navigation

Data visualizer from SaveEcoBot. Calculate AQI

Читати Українською 🇺🇦

SWUbanner

This software sets up and runs a nice dashboard with metrics from the CSV file and calculate AQI for PM2.5 and PM10.

Air Quality Index calculations are based on this document.

How it looks like

More screenshots and usage examples here.


MENU


Required software

Usage

Quick start

  1. Clone this repo
git clone [email protected]:MaxymVlasov/eco-data-visualizer.git

or download it as zip and unpack.

  1. Choose SaveEcoBot station on this map and click 'Details'
    map-details.png

  2. On the bottom you'll see Download raw data (CSV)
    download-csv.png
    click on the link and save CSV-file to ./data/original_data/ inside downloaded repo.

  3. Open the terminal in the root of eco-data-visualizer and run:

# Data preparation
docker build -t data-transformer ./data-transformer-app
docker run -v "$PWD"/data/:/app/data/ --rm data-transformer
# Run Grafana and DBs
docker-compose up -d
# Add data of sensors to InfluxDB
docker build -t add_influx_data ./provisioning/influx
docker run -v "$PWD"/data/influx/:/influx-data/ --rm --network=eco-data-visualizer_default add_influx_data

Depending on your internet bandwidth, CPU, Storage I/O, CSV file size, and the number of processed files First Init may take different times.
For example, in laptop with 100Mbit/s bandwidth, Intel Core i7-8550U (max clock speed 4Ghz), SSD disk and:

- 2 CSV files (together: 620MB) it takes 8m11s (6m09s to transform data)
- 1 CSV file (513MB) - 5m27s (3m47s to transform data)
- 1 CSV file (107MB) - 3m42s (2m to transform data)

  1. Open http://localhost/ for see visualizations!

Daily usage

Start

To start visualization open terminal in the root of the repo and run:

docker-compose up -d

Then open http://localhost/ to see visualizations.

Stop

To stop visualization open terminal in the root of the repo and run:

docker-compose stop

Sensors data

Process new data

  1. Download CSV file from SaveEcoBot station
  2. Move it to data/original_data folder in the repo.
  3. Run:
# Remove temporary files
docker run -v "$PWD"/data/:/app/ --rm amancevice/pandas:1.0.3-alpine sh -c "rm -f /app/csv/*.csv /app/influx/*.influx"
# Data preparation
docker build -t data-transformer ./data-transformer-app
docker run -v "$PWD"/data/:/app/data/ --rm data-transformer

Add new data

To add new data, open terminal in the root of the repo and run:

# Start services
docker-compose up -d
# Add new data
docker build -t add_influx_data ./provisioning/influx
docker run -v "$PWD"/data/influx/:/influx-data/ --rm --network=eco-data-visualizer_default add_influx_data

Remove data

To remove sensors data, open terminal in the root of the repo and run:

docker-compose down
docker volume rm eco-data-visualizer_sensors-data

Full cleanup

To clean up, open terminal in the root of the repo and run:

# Stop services
docker-compose down
# Remove volumes with settings and sensors data
docker volume rm eco-data-visualizer_grafana-settings eco-data-visualizer_sensors-data
# Remove temporary files
docker run -v "$PWD"/data/:/app/ --rm amancevice/pandas:1.0.3-alpine sh -c "rm -f /app/csv/*.csv /app/influx/*.influx"

For contributors

Use same instructions as in Quick start section but in step 4 add --build-arg ENV=dev to data-transformer-app.

docker build --build-arg ENV=dev -t data-transformer ./data-transformer-app

Future plans

  • Add Ukrainian localization
    • Code and message dashboard localization
    • Docs localization
  • In Grafana Create personal graphs for each sensor with own good-bad color limits and so on as for AQI
  • Grab exist metrics from 'phenomenon' colum, use SENSORS content only for user friendly names and localization
  • Add AQI support for everything specified in the doc
  • Optimize data-transformer-app
    • Parallel sensors operation execution
    • Use less Disk I/O operations

Want to help?

You can:

  • Improve this software (see Future plans section).
  • Donate to SaveEcoBot.
  • Assemble or buy Air quality monitoring station and connect it to SaveEcoBot. SaveDnipro can assemble and connect it for you. Buy here.

License and Copyrights

This software is licensed by Apache License 2.0.

All data from SaveEcoBot is licensed by Creative Commons Attribution License 4.0 International.

Other data and sources can be licensed in a different way.

Also, by using the code provided in this repository you agree with the following: