Dataflow Template for Firestore to Firestore Loadtest (Clone of PR 3098)

[jingqizz]

The loadtest-relevant scripts cloned from PR #3098

This PR depends on and includes all changes from PR #3392. Do not merge until #3392 is merged.

(cc @pacoavila808)

[gemini-code-assist]

Summary of Changes

Hello @jingqizz, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new Dataflow template that facilitates the batch transfer of data from one Firestore database to another. It includes robust resource management utilities for Firestore, a complete load testing framework to ensure performance and reliability, and the necessary Beam transforms to efficiently handle Firestore data operations. The changes also incorporate a Terraform module for simplified deployment, enhancing the template's usability and integration capabilities.

Highlights

• New Dataflow Template: Introduced a new Dataflow Flex Template named 'Firestore to Firestore' designed for batch copying documents between two Firestore databases. This template supports specifying source and destination projects/databases, as well as optional collection IDs and read time.
• Firestore Resource Managers: Added dedicated resource managers (FirestoreAdminResourceManager and FirestoreResourceManager) within the it-google-cloud-platform module to simplify the creation, management, and cleanup of Firestore databases and collections for integration testing purposes.
• Load Testing Framework: Included a comprehensive load testing setup for the new Firestore to Firestore template, featuring a FirestoreLoadGeneratorTransform to simulate data writes and a FirestoreToFirestoreLoadTest to evaluate performance under various conditions.
• Core Transforms for Firestore Operations: Implemented key Apache Beam transforms (CreatePartitionQueryRequestFn, PrepareWritesFn, RunQueryResponseToDocumentFn) to handle Firestore-specific operations like creating partitioned queries, preparing documents for writes, and extracting documents from query responses.
• Terraform Support: Provided an autogenerated Terraform configuration (dataflow_job.tf) to enable declarative deployment and management of the 'Firestore to Firestore' Dataflow job.

Changelog

• it/google-cloud-platform/pom.xml
  • Updated google-cloud-spanner dependency version.
  • Added new dependencies for google-cloud-pubsublite, google-cloud-firestore, and google-cloud-firestore-admin.
• it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/firestore/FirestoreAdminResourceManager.java
  • Added new class FirestoreAdminResourceManager for managing Firestore administrative tasks like creating and deleting databases.
• it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/firestore/FirestoreAdminResourceManagerException.java
  • Added new class FirestoreAdminResourceManagerException for custom exceptions related to Firestore admin resource management.
• it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/firestore/FirestoreResourceManager.java
  • Added new class FirestoreResourceManager for managing Firestore data operations such as writing, reading, and cleaning up collections.
• it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/firestore/FirestoreResourceManagerException.java
  • Added new class FirestoreResourceManagerException for custom exceptions related to Firestore data resource management.
• it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/firestore/package-info.java
  • Added package-level documentation for Firestore resource management classes.
• v2/firestore-to-firestore/README_Cloud_Firestore_to_Firestore.md
  • Added a new README file detailing the Firestore to Firestore template, including parameters, getting started instructions, and Terraform usage.
• v2/firestore-to-firestore/pom.xml
  • Added a new Maven module firestore-to-firestore with dependencies for the Dataflow template and testing utilities.
• v2/firestore-to-firestore/src/main/java/com/google/cloud/teleport/v2/templates/FirestoreToFirestore.java
  • Added the main class for the Firestore to Firestore Dataflow template, implementing the pipeline logic for reading from a source Firestore and writing to a destination Firestore.
• v2/firestore-to-firestore/src/main/java/com/google/cloud/teleport/v2/templates/package-info.java
  • Added package-level documentation for the Firestore to Firestore template.
• v2/firestore-to-firestore/src/main/java/com/google/cloud/teleport/v2/transforms/CreatePartitionQueryRequestFn.java
  • Added new transform CreatePartitionQueryRequestFn to generate Firestore PartitionQueryRequest objects from collection IDs.
• v2/firestore-to-firestore/src/main/java/com/google/cloud/teleport/v2/transforms/PrepareWritesFn.java
  • Added new transform PrepareWritesFn to convert Firestore Document objects into Write requests for the destination database, adjusting document names as needed.
• v2/firestore-to-firestore/src/main/java/com/google/cloud/teleport/v2/transforms/RunQueryResponseToDocumentFn.java
  • Added new transform RunQueryResponseToDocumentFn to extract Document objects from Firestore RunQueryResponse objects.
• v2/firestore-to-firestore/src/main/java/com/google/cloud/teleport/v2/transforms/package-info.java
  • Added package-level documentation for the Firestore-related transforms.
• v2/firestore-to-firestore/src/test/java/com/google/cloud/teleport/v2/templates/FirestoreLoadGeneratorTransform.java
  • Added new class FirestoreLoadGeneratorTransform to generate synthetic write load on a Firestore database for testing purposes.
• v2/firestore-to-firestore/src/test/java/com/google/cloud/teleport/v2/templates/FirestoreToFirestoreLoadTest.java
  • Added new load test class FirestoreToFirestoreLoadTest to evaluate the performance of the Firestore to Firestore template.
• v2/firestore-to-firestore/src/test/java/com/google/cloud/teleport/v2/transforms/CreatePartitionQueryRequestFnTest.java
  • Added unit tests for the CreatePartitionQueryRequestFn transform.
• v2/firestore-to-firestore/src/test/java/com/google/cloud/teleport/v2/transforms/PrepareWritesFnTest.java
  • Added unit tests for the PrepareWritesFn transform.
• v2/firestore-to-firestore/src/test/java/com/google/cloud/teleport/v2/transforms/RunQueryResponseToDocumentFnTest.java
  • Added unit tests for the RunQueryResponseToDocumentFn transform.
• v2/firestore-to-firestore/terraform/Cloud_Firestore_to_Firestore/dataflow_job.tf
  • Added Terraform configuration for deploying the Cloud Firestore to Firestore Dataflow Flex Template.
• v2/pom.xml
  • Added firestore-to-firestore as a new module to the parent pom.xml.

Activity

• This pull request is a clone of PR Dataflow Template for Firestore to Firestore Documents Copy #3098, indicating a re-submission or re-evaluation of previously proposed changes.
• The changes are explicitly stated to depend on PR Dataflow Template for Firestore to Firestore Documents Copy (Clone of PR 3098) #3392, suggesting a sequential integration plan where Dataflow Template for Firestore to Firestore Documents Copy (Clone of PR 3098) #3392 must be merged first.
• No specific review comments or further activity beyond the initial submission details are available in the provided context.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 0% with 138 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.86%. Comparing base (76ecc67) to head (0e2bff9).
⚠️ Report is 43 commits behind head on main.

Files with missing lines	Patch %	Lines
...eam/it/gcp/firestore/FirestoreResourceManager.java	0.00%	82 Missing ⚠️
...t/gcp/firestore/FirestoreAdminResourceManager.java	0.00%	56 Missing ⚠️

❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3403      +/-   ##
============================================
+ Coverage     51.24%   55.86%   +4.61%     
+ Complexity     5288     1767    -3521     
============================================
  Files          1001      498     -503     
  Lines         60954    28106   -32848     
  Branches       6682     2988    -3694     
============================================
- Hits          31238    15701   -15537     
+ Misses        27530    11455   -16075     
+ Partials       2186      950    -1236     

Components	Coverage Δ
spanner-templates	71.96% <ø> (+0.81%)	⬆️
spanner-import-export	∅ <ø> (∅)
spanner-live-forward-migration	79.85% <ø> (-0.02%)	⬇️
spanner-live-reverse-replication	77.33% <ø> (-0.02%)	⬇️
spanner-bulk-migration	87.99% <ø> (-0.02%)	⬇️
gcs-spanner-dv	85.73% <ø> (-0.61%)	⬇️

Files with missing lines	Coverage Δ
...t/gcp/firestore/FirestoreAdminResourceManager.java	0.00% <0.00%> (ø)
...eam/it/gcp/firestore/FirestoreResourceManager.java	0.00% <0.00%> (ø)

... and 532 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.