Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] expect_table_row_count_to_equal should return the actual row count as well as the expression #321

Open
siljamardla opened this issue Nov 1, 2024 · 0 comments

Comments

@siljamardla
Copy link

siljamardla commented Nov 1, 2024

I want to monitor some row counts on my table. I'm planning to set up row count tests and enable storing failures.
I plan to do this with multiple different conditions. I actually expect some of these two fail. The aim is to monitor how badly they fail. The idea is to visualise the outcome in a calendar heat map, to give me a visual on when the issues are big.

I could use the row count test, configured like this:

      - dbt_expectations.expect_table_row_count_to_equal:
          value: 0
          group_by: [created_date]
          row_condition: some_column='failed'
          config:
            severity: warn
            store_failures: true

However, the outcome is a true/false only.

Sample compiled code:

    with grouped_expression as (
    select
        created_date as col_1,
        count(*) = 0 as expression
    from 
      schema_name.table_name
    where
       some_column='failed'
    group by  created_date
),
validation_errors as (
    select *
    from
        grouped_expression
    where
        not(expression = true)
)
select *
from validation_errors
image

I wish the compiled code would have an additional column for the actual row_count, like this:
Sample compiled code:

    with grouped_expression as (
    select
        created_date as col_1,
        count(*) as row_count,
        count(*) = 0 as expression
    from 
      schema_name.table_name
    where
       some_column='failed'
    group by  created_date
),
validation_errors as (
    select *
    from
        grouped_expression
    where
        not(expression = true)
)
select *
from validation_errors

to return
image

It's not a breaking change as the logic of the test remains the same. We're just making the stored output more informative.

I'm happy to contribute, but I could not immediately figure out where to make this change. The actual code compiling happens through so many macros that I'm not (yet) familiar with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant