soda-data-quality-checks

Demo: Soda data quality checks with LocalStack Snowflake

This project illustrates how to use the Soda framework to run data quality checks against Snowflake tables, entirely on your local machine.

The code is based on the Snowflake Quickstart Guide on Data Quality Testing with Soda.

Prerequisites

• A valid LocalStack for Snowflake license. Your license provides a LOCALSTACK_AUTH_TOKEN.
• localstack CLI with LOCALSTACK_AUTH_TOKEN environment variable set
• awslocal CLI
• LocalStack Snowflake emulator
• snow CLI with localstack connection profile pointing to LocalStack Snowflake

To install the snow CLI with a local connection profile, follow the instructions here. It basically boils down to running this command:

snow connection add \
    --connection-name localstack \
    --user test \
    --password test \
    --account test \
    --host snowflake.localhost.localstack.cloud

Instructions

Install dependencies

Run the following command to install dependencies for the project (into a local Python virtual environment):

make install

Start LocalStack

To start LocalStack with custom Snowflake logging flags, use:

DOCKER_FLAGS='-e SF_LOG=trace' \
  DEBUG=1 \
  localstack start -s snowflake -d

Initialize the data tables

The sample application provides Makefile targets to simplify the setup process.

Run the following command to initialize the environment and seed test data into local Snowflake:

make init

Use Soda to run data quality checks

Once the test data has been set up, we can run the Soda data quality checks via this command:

make scan

License

The code in this project is licensed under the Apache 2.0 License.