How to Validate Data Diffs in the Extract + Load Process with Asset Checks

[sungchun12]

Hey There!

I've been working with @tacastillo to build out a simple integration how-to on using open source data-diff to solve a problem in data replication: proving data is the same (think: postgres <> snowflake replicated via fivetran)

This makes most sense in a guide here as it's related to an open source library : https://docs.dagster.io/integrations

Example tested by Tim! https://gist.github.com/tacastillo/bcf457a14fc70701825e410581c3eee9

I want to make it dead simple to trust data before end users yell at data teams for easily preventable errors.

[PedramNavid]

Hi @sungchun12 ! The Guides are meant for existing Dagster Libraries, e.g. dagster-dbt or dagster-fivetran. We don't currently have a dagster-data-diff library although if you're open to writing one, I'd be happy to co-ordinate on that?

Outside of the integration, if you'd like we can also pair on a blog post together. Feel free to reach out to me on the Dagster Slack and we can decide on the most appropriate path

[sungchun12]

Hey Pedram!

Thanks for the clarity there. I'll DM you on Dagster slack to understand what it takes to build out a Dagster library and whether it makes sense in the short vs. long term.

And yes, happy to blog post together too!