Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SNOW-1566362] Add sql counter check for snowpark python test_cte #2370

Merged
merged 8 commits into from
Oct 2, 2024
Merged

[SNOW-1566362] Add sql counter check for snowpark python test_cte #2370

merged 8 commits into from
Oct 2, 2024

Conversation

sfc-gh-yzou
Copy link
Collaborator

@sfc-gh-yzou sfc-gh-yzou commented Sep 30, 2024 •
edited
Loading

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

SNOW-1566362

  1. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.

  2. Please describe how your code solves the related issue.

  1. Move sql counter check infrastructure from modin to snowpark in general
  2. applies sql counter check to test_cte.py

@sfc-gh-yzou sfc-gh-yzou added the NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md label Sep 30, 2024
@sfc-gh-yzou
Copy link
Collaborator Author

snowpark pandas daily job started https://ci-dev-142.int.snowflakecomputing.com/job/SnowparkPythonSnowPandasDailyRegressRunner/874/

@sfc-gh-yzou
Copy link
Collaborator Author

daily job started https://github.com/snowflakedb/snowpark-python/actions/runs/11115145384

@sfc-gh-yzou sfc-gh-yzou marked this pull request as ready for review September 30, 2024 22:32
@sfc-gh-yzou sfc-gh-yzou requested review from a team as code owners September 30, 2024 22:32
sfc-gh-aalam
sfc-gh-aalam reviewed Sep 30, 2024
View reviewed changes
tests/integ/utils/sql_counter.py Outdated
sql_count_records: dict[str, dict[str, list[dict[str, Optional[Scalar]]]]]
sql_count_records: Dict[str, Dict[str, List[Dict[str, Optional[Scalar]]]]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im not sure if this change is actually needed.

Copy link
Collaborator Author

@sfc-gh-yzou sfc-gh-yzou Sep 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that is needed, otherwise it will give an error "type" is not subscritble, it is because the python version restriction, snowpark pandas requires python 3.9 +, so have no problem with using dict, or list, but snowpark python still allows 3.8

sfc-gh-aalam
sfc-gh-aalam approved these changes Sep 30, 2024
View reviewed changes
tests/integ/test_cte.py
@@ -100,6 +104,12 @@ def count_number_of_ctes(query):
lambda x: x.to_df("a1", "b1").alias("L"),
],
)
@sql_count_checker(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about also adding describe_count here and other tests?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the describe count check is embeded in check_result, which is used in this test point already.
there is also a limitation that the sql_counter_check annotation currently seems not working with describe_count yet, a ticket has been created here to https://snowflakecomputing.atlassian.net/browse/SNOW-1704919 to track the issue

sfc-gh-evandenberg
sfc-gh-evandenberg approved these changes Sep 30, 2024
View reviewed changes
Copy link
Collaborator

@sfc-gh-evandenberg sfc-gh-evandenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sfc-gh-yzou sfc-gh-yzou force-pushed the yzou-SNOW-1566362-sql-counter-tests branch 2 times, most recently from 0e89c42 to c136db4 Compare October 1, 2024 17:54
@sfc-gh-yzou sfc-gh-yzou force-pushed the yzou-SNOW-1566362-sql-counter-tests branch from c136db4 to 8c7dd0d Compare October 1, 2024 22:15
@sfc-gh-yzou sfc-gh-yzou merged commit 32bfa1d into main Oct 2, 2024
35 of 36 checks passed
@sfc-gh-yzou sfc-gh-yzou deleted the yzou-SNOW-1566362-sql-counter-tests branch October 2, 2024 00:05
@github-actions github-actions bot locked and limited conversation to collaborators Oct 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@sfc-gh-aalam sfc-gh-aalam sfc-gh-aalam approved these changes

@sfc-gh-evandenberg sfc-gh-evandenberg sfc-gh-evandenberg approved these changes

@sfc-gh-aling sfc-gh-aling Awaiting requested review from sfc-gh-aling sfc-gh-aling is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-yuwang sfc-gh-yuwang Awaiting requested review from sfc-gh-yuwang sfc-gh-yuwang is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-jrose sfc-gh-jrose Awaiting requested review from sfc-gh-jrose sfc-gh-jrose is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-jdu sfc-gh-jdu Awaiting requested review from sfc-gh-jdu

Assignees
No one assigned
Labels
NO-CHANGELOG-UPDATES This pull request does not need to update CHANGELOG.md snowpark-pandas
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sfc-gh-yzou @sfc-gh-aalam @sfc-gh-evandenberg