Status of testing Providers that were prepared on July 09, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.26.0rc1

• Do not dynamically determine op links for emr serverless start job operator (#40627): @o-nikolas
  Linked issues:
  • Linked Issue #40103: @fjmacagno
• Be able to remove ACL in S3 hook's copy_object (#40518): @anteverse
• Fix OpenLineage extraction for deferrable AthenaOperator (#40545): @kacpermuda
• Reduce memory footprint of s3 key trigger (#40473): @dstandish
• Adding cluster to ecs trigger event to avoid defer error (#40482): @ellisms
  Linked issues:
  • Linked Issue #40207: @frantoja
• Use base aws classes in AWS Glue Data Catalog Sensors (#40492): @gopidesupavan
  Linked issues:
  • Linked Issue #35278: @Taragolis
• Use base aws classes in AWS Glue Crawlers Operators/Sensors/Triggers (#40504): @gopidesupavan
  Linked issues:
  • Linked Issue #35278: @Taragolis
• Share data loader to across asyncio boto sessions (#40658): @dstandish

Provider cncf.kubernetes: 8.3.3rc1

• Solve failing KPO task with task decorator and imported typing elements (#40642): @BenediktRamsauer
  Linked issues:
  • Linked Issue #40585: @BenediktRamsauer

Provider common.sql: 1.14.2rc1

• FIX: DbApiHook.insert_rows unnecessarily restarting connections (#40615): @dabla

Provider databricks: 6.7.0rc1

• Make Databricks operators' json parameter compatible with XComs, Jinja expression values (#40471): @boraberke
  Linked issues:
  • Linked Issue #35433: @josh-fell
• Bug/fix support azure managed identities in Databricks operator (#40332): @marcel-martinelli
  Linked issues:
  • Linked Issue #38762: @jtv8

Provider docker: 3.12.2rc1

• DockerOperator TaskFlow - correct arguments in python command (#39620): @antoinetavant
  Linked issues:
  • Linked Issue #39319: @antoinetavant
• Improve logging behavior of DockerOperator (#40489): @jscheffl

Provider fab: 1.2.1rc1

• Add backward compatibility to CSRF protection of /logout method (#40479): @shahar1
  Linked issues:
  • Linked Issue #40470: @Al2a2d2m

Provider google: 10.21.0rc1

• Update Google Cloud Generative Model Hooks/Operators to bring parity with Vertex AI API (#40484): @CYarros10
• DataflowStartFlexTemplateOperator. Check for Dataflow job type each check cycle. (#40584): @theopus
• Added chunk_size parameter to LocalFilesystemToGCSOperator (#40379): @jsjasonseba
• Add support for query parameters to BigQueryCheckOperator (#40556) (#40558): @aldenstpage
• DataprocCreateBatchOperator - Link for asynchronous mode (#40643): @kandharvishnu
• Revert "Fix GCSToGCSOperator behavior difference for moving single object (#40162)" (#40577): @MaksYermak
• fix BigQueryInsertJobOperator's return value and openlineage extraction in deferrable mode (#40457): @kacpermuda
• fix OpenLineage extraction for GCP deferrable operators (#40521): @kacpermuda
  Linked issues:
  • Linked Issue #40457: @kacpermuda
• 40559: fix add project_id to submit_batch_job hook (#40560): @guillaumeblaquiere
  Linked issues:
  • Linked Issue #40559: @guillaumeblaquiere
• Resolve google deprecations in tests (#40629): @dirrao
• Resolve google vertex ai deprecations in tests (#40506): @dirrao
• Add notes about reverting the change in GCSToGCSOperator (#40579): @potiuk

Provider influxdb: 2.6.0rc1

• add timeout to influx db (#40439): @romsharon98
• Fix failure of "Add connection" introduced by influxdb change (#40469): @potiuk
  Linked issues:
  • Linked Issue #40439: @romsharon98

Provider microsoft.azure: 10.2.0rc1

• S3 to wasb (#40511): @jroachgolf84

Provider openlineage: 1.9.1rc1

• openlineage: fix parsing dag tree with MappedOperator (#40621): @kacpermuda

Provider oracle: 3.10.3rc1

• Warn user about unused replace parameter in insert_rows (OracleHook) (#39408): @RichieSK

Provider snowflake: 5.6.0rc1

• chore: Add param support for client_request_mfa_token (#40394): @vanducng

Provider teradata: 2.4.0rc1

• Added support of Teradata Compute Cluster Provision, Decommission, Suspend and Resume operations (#40509): @sc250072

Provider weaviate: 2.0.0rc1

• Upgrade to weaviate-client to v4 (#40194): @Lee-W
• Fix mypy problems in new waeviate client (#40330): @potiuk
• Update pandas minimum requirement for Python 3.12 (#40272): @potiuk
• Add dependency to httpx >= 0.25.0 everywhere (#40256): @potiuk

Provider ydb: 1.1.0rc1

• support auth key from content and from file (#40390): @uzhastik

All users involved in the PRs:
@BenediktRamsauer @marcel-martinelli @kacpermuda @uzhastik @jsjasonseba @antoinetavant @theopus @CYarros10 @sc250072 @MaksYermak @boraberke @guillaumeblaquiere @o-nikolas @Lee-W @shahar1 @potiuk @ellism

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

I checked all my changes they look good. One SMALL thing.. I noticed that weaviate changelog is wrong - the 2.0.0 changelog contains also 1.4.2 changes and 1.4.2 is missing from changelog. This is caused by #40194 - where 2.0.0 description replaced 1.4.2 one rather than added it. This should not change the release status of 2.0.0 because changelog is not part of the package and we can fix it by cherry-picking #40663 that fixes it and reganerating the documentation (cc: @eladkal )

[kacpermuda]

I checked all my changes they look good. Thanks !

[sc250072]

I checked Teradata airflow provider and new feature(s) are working as expected. Thank you!.

[MaksYermak]

I have checked all my changes they work as expected.

[vanducng]

I checked my change for the snowflake provider, it works as expected.

[eladkal]

Weaviate provider will have rc2

[marcel-martinelli]

I checked all my changes on the Databricks operator; it works fine!

[gopidesupavan]

Verified #40492 and #40504 are working fine. Thank you for the release efforts.

[dirrao]

I have verified #40629 and #40506. They are working as expected.

[antoinetavant]

I have verified the Docker operator, both #39620 and #40489 works as expected on airflow:2.8.3-python3.11 as well as airflow:2.9.2-python3.11

[BenediktRamsauer]

I have tested #40642 and it is working.

[guillaumeblaquiere]

Ok for #40560

[uzhastik]

OK for ydb provider and #40390

[potiuk]

I am not sure if that qualifies for RC2, but currently in main AWS auth manager fails compatibility test for 2.8 - i described it in #40684 - cc: @vincbeck, @eladkal

[potiuk]

So I guess with #40690 it means RC2 for Amazon @vincbeck ?

[vincbeck]

Yes!

[eladkal]

You wish to exclude fab or amazon from rc1?

Amazon provider will be excluded from rc1

[anteverse]

#40518

Checked and working as expected.

[shahar1]

#40479 is ok

[jscheffl]

I just found a serious problem in databricks provider. PR coming in a moment. If PR 40153 is contained in the release then for sure we need to stop and a -1 from my side. Sent from Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Aurelien Didier ***@***.***> Sent: Wednesday, July 10, 2024 11:12:28 PM To: apache/airflow ***@***.***> Cc: Scheffler Jens (XC-AS/EAE-ADA-T) ***@***.***>; Mention ***@***.***> Subject: Re: [apache/airflow] Status of testing Providers that were prepared on July 09, 2024 (Issue #40661) #40518<#40518> Checked and working as expected. — Reply to this email directly, view it on GitHub<#40661 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWVTFDO7LONFW5AE4BDKVGLZLWPTZAVCNFSM6AAAAABKSHNSR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRGQ4DEOBUGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jscheffl]

Dont see my previous email, -1 for databricks, raised a fix PR #40708 but would rather recomment to roll back #40153 Sent from Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Elad Kalif ***@***.***> Sent: Wednesday, July 10, 2024 6:32:10 PM To: apache/airflow ***@***.***> Cc: Scheffler Jens (XC-AS/EAE-ADA-T) ***@***.***>; Mention ***@***.***> Subject: Re: [apache/airflow] Status of testing Providers that were prepared on July 09, 2024 (Issue #40661) I will exclude fab from rc1 — Reply to this email directly, view it on GitHub<#40661 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AWVTFDLUQO5ARSR2PFJPOR3ZLVOYVAVCNFSM6AAAAABKSHNSR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQHE3TMNRVGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jscheffl]

Ah, emails are getting to ticket and not to devlist :-D

As far as I see the PR #40153 is not in the providers-cut? Then it is OK...

[pankajkoti]

yes, #40153 is not in this providers wave

[dabla]

PR 40615 partially fixes the issue, it doesn't fix it for the JdbcHook as there the driver_path and driver_class property causes the same issue as placeholder did. Also mentioned this in original issue.

[potiuk]

PR 40615 partially fixes the issue, it doesn't fix it for the JdbcHook as there the driver_path and driver_class property causes the same issue as placeholder did. Also mentioned this in original issue.

Yeah. Good catch @dabla -> I think it's not a blocker for this release, it's not a regression comparing to already released common.sql and we can always fix it in the next round (and it has rather small-ish impact).

[eladkal]

@dabla can you raise PR with a fix for it?

[dabla]

@dabla can you raise PR with a fix for it?

Hey Elad, yes I already have a fix for it, but would like to see PR #40665 merged first as this will allow me to have a cleaner solution. Unless you want this fix first then I will have to refactor the fix in #40665 afterwards, that's also possible. It's up to you

[potiuk]

I think it's not a regression and can be easily fixed in the next round if you ask me :) - there are few other things that @dabla is contributing in this area #40665 and likely they could be done in next release ?

[potiuk]

Similar answer :)

[dabla]

Same for me, if you want the fix @eladkal, it can be done now/today...

[dabla]

I already started with the PR for this fix, can merge #40665 afterwards, not such a big issue anyway

[eladkal]

Thank you everyone. Providers are released (excluding amazon and weaviate)

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[dabla]

@dabla can you raise PR with a fix for it?

#40751