Skip to content

Automate and streamline the alerting / notification process for dbt test results๐Ÿž๐Ÿš€

License

Notifications You must be signed in to change notification settings

infinitelambda/diqu

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 

History

71 Commits
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

diqu

Documentation PyPI version License: Apache 2.0 python-cli python codecov

Automate and streamline the alerting/ notification process for dbt test results using this versatile CLI companion tool. Receive detailed alerts & test metadata seamlessly on various platforms, promoting improved collaboration on dbt project issues ๐Ÿž๐Ÿš€.

Who is this for

This tool is designed for individuals or teams seeking to automate the management of their dbt project issues (test warnings, errors... etc) outside the dbt environment.

Features

  • Automated alerts and notifications based on recorded dbt test results.
  • Built-in support for dq-tools and custom query input.
  • Auto-labels deprecated tests for quick & easy identification.
  • Creates and updates Jira tickets with the latest tests' metadata.
  • Sends succinct and informative messages to a dedicated Slack channel.

dbt alert rant ๐ŸŸข ๐ŸŸก ๐Ÿ”ด โšซ

"Run errors are red,
Test warnings are yellow,
Where's my bug alert,
And joint bugfix workflow?"

-- by diqu OG Contributors

diqu Concept

Supported Modules

  • Sources (DWH connections)
    • Snowflake
    • CSV file
  • Package (parsing dbt test results)
    • dq-tools
    • Custom query
  • Alert Modules (alert/ notification)
    • Jira
    • Slack

Installation

pip install diqu [--upgrade]

๐Ÿ““ NOTE: The required Data Warehouse (DWH) module should already be installed if you are using diqu in a working dbt project. If not, please perform additional steps to install these DWH modules. For example, if you're using Snowflake:

pip install "snowflake-connector-python[pandas]"
pip install "snowflake-connector-python[secure-local-storage]"

Usage

preflight
# define the query params
export ISSUE_DEPRECATED_WINDOW_IN_DAYS=your_issue_deprecation_time_in_day, default to "3"
export ISSUE_UPDATE_WINDOW_IN_DAYS=your_issue_historical_data_update_window_in_days, default to "14"

# build dq-tools log table
dbt run -s dq_tools
diqu alert --to slack --to jira
04:33:17  diqu: INFO - Run with diqu==1.0.0 ๐Ÿƒ
04:33:19  diqu: INFO - Using dbt project at: /path/to/dbt/project
04:33:19  diqu: INFO - Using dbt profiles.yml at: ~/.dbt
04:33:19  diqu: INFO - Using snowflake connection
04:33:19  diqu: INFO - Looking for the query in: ./dq_tools__get_test_results.sql
04:33:23  diqu: INFO - Alerting to: SLACK
04:33:23  diqu: INFO - โœ… Done > Slack
04:33:23  diqu: INFO - Alerting to: JIRA
04:33:23  diqu: INFO - โœ… Done > JIRA

Alert Modules Configurations

Slack

  • Use the environment variables to configure the Slack Channel:

    export SLACK_TOKEN=your_token
    export SLACK_CHANNEL=your_channel_name
    
    diqu alert --to slack

Jira Board

  • Use the environment variables to configure the JIRA Board:

    export JIRA_SERVER=your_jira_server e.g. https://your_value.atlassian.net/
    export JIRA_AUTH_USER=your_service_account e.g. dqt_user@your_value.com
    export JIRA_AUTH_PASSWORD=your_service_token e.g. ATATTxxxxx
    export JIRA_PROJECT_ID=your_project_id e.g. 106413
    export JIRA_ISSUE_TYPE=your_issue_type, default to "Bug"
    export JIRA_OPEN_ISSUES_FILTER_BY_SUMMARY=your_issue_filter_on_title, default to "dq-tools"
    
    diqu alert --to jira

๐Ÿ“– For more details, please visit the documentation site.

Quick Demo

Watch the video

How to Contribute

This Auto Alert (diqu) tool is an open-source software. Whether you are a seasoned open-source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.

๐Ÿ‘‰ See CONTRIBUTING guideline for more details or check out CONTRIBUTING.md

๐ŸŒŸ And then, kudos to our beloved Contributors:

About Infinite Lambda

Infinite Lambda is a cloud and data consultancy. We build strategies, help organizations implement them, and pass on the expertise to look after the infrastructure.

We are an Elite Snowflake Partner, a Platinum dbt Partner, and two-times Fivetran Innovation Partner of the Year for EMEA.

Naturally, we love exploring innovative solutions and sharing knowledge, so go ahead and:

๐Ÿ”ง Take a look around our Git

โœ๏ธ Browse our tech blog

We are also chatty, so:

๐Ÿ‘€ Follow us on LinkedIn

๐Ÿ‘‹๐Ÿผ Or just get in touch

About IL