Status of testing Providers that were prepared on October 27, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 9.1.0rc1

• feat: add OpenLineage support for RedshiftToS3Operator (#41632): @kacpermuda
  Linked issues:
  • Linked Issue #41575: @Artuz37
• Add SageMakerProcessingSensor (#43144): @vVv-AA
• Make RedshiftDataOperator handle multiple queries (#42900): @jroachgolf84
• Remove sqlalchemy-redshift dependency (#43271): @mobuchowski
• feat(providers/amazon): Use asset in common provider (#43110): @Lee-W
• Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954): @gopidesupavan

Provider apache.beam: 5.9.0rc1

• Add early job_id xcom_push for google provider Beam Pipeline operators (#42982): @olegkachur-e

Provider apache.druid: 3.12.0rc1

• Add possibility to override the conn type for Druid (#42793): @Rasnar

Provider apache.hdfs: 4.6.0rc1

• added MultipleFilesWebHdfsSensor (#43045): @eilon246810
• Fixed failing static checks & provider tests (#43122): @kaxil

Provider apache.livy: 3.9.2rc1

• Add lower bound to asgiref in providers (#43001): @rawwar

Provider apache.spark: 4.11.2rc1

• Changed conf property from str to dict in SparkSqlOperator (#42835): @duhizjame

Provider cloudant: 4.0.2rc1

• More correctly work around uv sync issue with python 3.10+ only dep (#43227): @ashb
• Update cloudant versions to be py-version aware so uv lock can work. (#43217): @ashb
  Linked issues:
  • Linked Issue #41555: @topherinternational
  • Linked Issue #21004: @uranusjr

Provider cncf.kubernetes: 9.0.1rc1

• (fix): HybridExecutor tasks of other executor rescheduled in kubernetes executor (#43003): @pavansharma36
  Linked issues:
  • Linked Issue #42151: @iw-pavan
• fix: use instance base_container_name to fetch logs on trigger_reentry (#42960): @peloyeje
• Add kubernetes_conn_id to templated fields (#42786): @gopidesupavan
  Linked issues:
  • Linked Issue #42546: @nikhilkarve
• ✨ Allow node_selector templating in KPO (#43051): @bdsoha
• Adding support for Kubernetes package version 31.0.0 (#42907): @dirrao

Provider common.sql: 1.19.0rc1

• Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357): @dabla
• Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891): @dabla
• SqlSensor enhancement: #37437 (#43107): @vVv-AA
  Linked issues:
  • Linked Issue #37437: @amolsr
• fix mypy failure(provider) in SqlSensor (#43206): @rawwar

Provider databricks: 6.12.0rc1

• Add TimeoutError to be a retryable error in databricks provider (#43137): @rawwar
  Linked issues:
  • Linked Issue #43128: @rawwar
• Add ClientConnectorError to be a retryable error in databricks provider (#43091): @rawwar
• DatabricksHook: fix status property to work with ClientResponse used in async mode (#43333): @lucafurrer
  Linked issues:
  • Linked Issue #43269: @lucafurrer
• [DatabricksHook] Respect connection settings (#42618): @xitep

Provider dbt.cloud: 3.11.1rc1

• Add lower bound to asgiref in providers (#43001): @rawwar

Provider fab: 1.5.0rc1

• feat(providers/fab): Use asset in common provider (#43112): @Lee-W
• Fix revoke Dag stale permission on airflow < 2.10 (#42844): @joaopamaral
  Linked issues:
  • Linked Issue #42743: @RostD
• Bump Flask-AppBuilder to 4.5.2 (#43309): @kaxil
• Upgrade FAB to 4.5.1 (#43251): @potiuk
• Move user and roles schemas to fab provider (#42869): @vincbeck
• Move the session auth backend to FAB auth manager (#42878): @vincbeck
  Linked issues:
  • Linked Issue #42634: @vincbeck

Provider google: 10.25.0rc1

• feat: add Hook Level Lineage support for GCSHook (#42507): @kacpermuda
  Linked issues:
  • Linked Issue #40819: @mobuchowski
• feat: sensor to check status of Dataform action (#43055): @steve148
• Create Operators for Google Cloud Vertex AI Context Caching (#43008): @CYarros10
• Fix outdated CloudRunExecuteJobOperator docs (#43195): @p13rr0m
• Fix TestTranslationLegacyModelPredictLink dataset_id error for google provider translation operator (#42463): @olegkachur-e
• Add a debug log for creating batch workloads in dataproc (#43265): @romsharon98
• add min version to plyvel (#43129): @rawwar
• vertex ai training operators: add display_name to rendered fields (#43028): @WSHoekstra
  Linked issues:
  • Linked Issue #43027: @WSHoekstra
• Make google provider pyarrow dependency explicit (#42996): @saucoide
  Linked issues:
  • Linked Issue #42924: @saucoide
• Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954): @gopidesupavan

Provider http: 4.13.2rc1

• fix: HttpSensorTrigger to include method when serializing (#42925): @rawwar
• Use url_from_endpoint inside HttpHook. (#42785): @simi
• add lowerbound to requests-toolbelt and replace requests_toolbelt with requests-toolbelt (#43020): @rawwar
  Linked issues:
  • Linked Issue #42989: @potiuk
• Add lower bound to asgiref in providers (#43001): @rawwar

Provider jenkins: 3.7.2rc1

• Remove unnecessary return value from jenkins_request_with_headers (#43207): @romsharon98

Provider microsoft.azure: 11.0.0rc1

• Add context to Azure Service Bus Message callback (#43370): @perry2of5
• Feature: Added event_handler parameter in MSGraphAsyncOperator (#42539): @dabla
• Add documentation for the PowerBIDatasetRefresh Operator. (#42754): @ambika-garg
• Add upperbound to microsoft-kiota-abstractions (#43021): @rawwar
• Restrict looker-sdk version 24.18.0 and microsoft-kiota-http 1.3.4 (#42954): @gopidesupavan

Provider mongo: 4.2.2rc1

• fix(providers/mongo): prevent applying lower method on bool field (#43024): @josix
  Linked issues:
  • Linked Issue #42930: @FarouziAbir

Provider mysql: 5.7.3rc1

• Improve the cursor type definition of mysql (#43376): @yangyulely

Provider odbc: 4.8.0rc1

• Also use ODBC connection for sqlalchemy engine in OdbcHook like JdbcHook (#43145): @dabla

Provider openlineage: 1.13.0rc1

• feat(providers/openlineage): Use asset in common provider (#43111): @Lee-W
• Ignore attr-defined for compat import (#43301): @uranusjr
  Linked issues:
  • Linked Issue #43142: @Lee-W
• nit: remove taskgroup's tooltip from OL's AirflowJobFacet (#43152): @kacpermuda
• require 1.2.1 common.compat for openlineage provider (#43039): @mobuchowski

Provider oracle: 3.12.0rc1

• Add sequence insert support to OracleHook (#42947): @Lee2532

Provider slack: 8.9.1rc1

• adding support for snippet type in slack api (#43305): @Bowrna
• passing the filetype for SlackAPIFileOperator (#43069): @Bowrna
  Linked issues:
  • Linked Issue #42889: @hzlmn

Provider ssh: 3.14.0rc1

• SSHHook expose auth_timeout parameter (#43048): @gyandeeps

All users involved in the PRs:
@steve148 @gyandeeps @mobuchowski @gopidesupavan @uranusjr @dabla @kacpermuda @eilon246810 @kaxil @ambika-garg @Bowrna @joaopamaral @pavansharma36 @rawwar @vVv-AA @bdsoha @olegkachur-e @p13rr0m @romshar

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Checked my changes - all look good !

[josix]

Checked mongo: 4.2.2rc1, it works as expected, thanks!

[bdsoha]

Checked cncf kubernetes/9.0.1rc1, looks OK!

[Lee2532]

checked oracle: 3.12.0rc1, it works, thanks!

[rawwar]

Checked following and don't see any issues.
#43001 , #43206, #43001 ,#43129 ,#42925 ,#43020 , #43001 ,#43021

[gyandeeps]

Checked ssh: 3.14.0rc1 and it looks good. Thanks.

[perry2of5]

Azure Service Bus message callbacks look good in 11.0.0rc1 (from PR #43370)

[p13rr0m]

Checked #43195. Looks good as well.

[xitep]

Verified #42618 locally with apache-airflow-providers-databricks==6.12.0rc1 ... works as expected for me

[saucoide]

checked #42996 / #42924

[yangyulely]

Checked mysql: 5.7.3rc1, all looks good.

[Lee-W]

amazon: 9.1.0rc1 and fab: 1.5.0rc1 do not work with airflow 2.10.x on dataset permissions.

corresponding PRs have been sent

#43470
#43469

[gopidesupavan]

Checked my changes all are okay!..

[eladkal]

I will exclude amazon and fab from this release

[vVv-AA]

Tested SqlSensor enhancement: #37437 (#43107) Looks ok.

…

On Wed, 30 Oct 2024, 12:35 Elad Kalif, ***@***.***> wrote: I will exclude amazon and fab from this release — Reply to this email directly, view it on GitHub <#43416 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADAMDO67YQGZYERCZYBRO4LZ6CAJ5AVCNFSM6AAAAABQV42RR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBWGAZDAMRSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dabla]

Will check mine tonight or tomorrow

[eladkal]

Thank you everyone. Providers are released.
fab and amazon providers are excluded from this wave

I invite everyone to help improve providers for the next release, a list of open issues can be found here.

[dabla]

43357
43145

A bit late, but works as expected