Cherrypick #37725

[Amar3tto]

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request, a cherry-pick from a previous change, focuses on refining the testing infrastructure for Apache Beam's Python SDK, particularly concerning machine learning components. It introduces specific pytest markers for Vertex AI embedding tests and updates the CI configuration to manage how these tests are executed. Additionally, it adjusts pyarrow versions in test suites and updates transformers dependencies, aiming to improve test efficiency and maintain compatibility with evolving library versions.

Highlights

• Vertex AI Test Tagging: Applied a new vertex_ai_postcommit pytest mark to various Vertex AI embedding tests to categorize them for specific CI runs.
• Pytest Exclusion Configuration: Updated tox.ini configurations to exclude tests marked with vertex_ai_postcommit from certain pytest execution environments, likely to optimize or manage CI test suites.
• Pyarrow Version Updates: Modified build.gradle to adjust the pyarrow versions used in Python 3.10 test suites, removing some older versions and adding newer ones.
• Transformers Test Environment Refinement: Refined the dependencies and configuration for py{310,311}-transformers test environments in tox.ini, including updates to transformers version ranges and pip_pre settings.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• sdks/python/apache_beam/ml/rag/embeddings/vertex_ai_test.py
  • Imported the pytest module.
  • Added the @pytest.mark.vertex_ai_postcommit decorator to the VertexAITextEmbeddingsTest class.
• sdks/python/apache_beam/ml/transforms/embeddings/vertex_ai_test.py
  • Imported the pytest module.
  • Added the @pytest.mark.vertex_ai_postcommit decorator to the VertexAIEmbeddingsTest class.
  • Added the @pytest.mark.vertex_ai_postcommit decorator to the VertexAIImageEmbeddingsTest class.
  • Added the @pytest.mark.vertex_ai_postcommit decorator to the VertexAIMultiModalEmbeddings class.
• sdks/python/test-suites/tox/py310/build.gradle
  • Removed pyarrow versions 9 through 14 from the py310 test suite.
  • Added pyarrow version 6 to the py310 test suite.
  • Added pyarrow versions 19 through 23 to the py310 test suite.
• sdks/python/tox.ini
  • Set pip_pre to False and simplified extras for py{310,311}-transformers test environments.
  • Updated the transformers version range for the 448 environment.
  • Modified the pytest command for apache_beam/ml/transforms/embeddings to exclude tests marked vertex_ai_postcommit.
  • Updated the run_pytest.sh command to exclude tests marked vertex_ai_postcommit for apache_beam/ml/transforms/embeddings.

Activity

• No human activity has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩