E2E GCS Sink additional test scenarios. #1478

bijay27bit · 2024-12-17T05:51:38Z

No description provided.

google-cla · 2024-12-17T05:51:42Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

itsmekumari · 2024-12-17T06:10:34Z

src/e2e-test/features/gcs/sink/GCSSink.feature

@@ -265,3 +265,175 @@ Feature: GCS sink - Verification of GCS Sink plugin
    Then Open and capture logs
    Then Verify the pipeline status is "Succeeded"
    Then Verify data is transferred to target GCS bucket
+
+#Added new scenarios for GCS Sink - Bijay


Remove the commented line here.

itsmekumari · 2024-12-17T06:11:18Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+
+#Added new scenarios for GCS Sink - Bijay
+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario:Validate successful records transfer from BigQuery to GCS with macro enabled at sink


Add the macro scenario in a separate feature file with name macro, refer other plugins feature file for naming convention.

itsmekumari · 2024-12-17T06:11:53Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST
+  Scenario Outline: To verify data is getting transferred successfully from BigQuery to GCS with contenttype selection


Add the validation for file format as well

In progress.

itsmekumari · 2024-12-17T06:12:15Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+      | tsv        | text/plain    |
+
+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field


Why are we adding macro here again, It is already covered in macro enabled scenario. It should be for without macro enabled

itsmekumari · 2024-12-17T06:12:37Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery


Use the latest existing steps. This is a common review comment across all scenarios.

itsmekumari · 2024-12-17T06:13:14Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Close the GCS properties
+    Then Save the pipeline
+    Then Preview and run the pipeline
+    Then Enter runtime argument value "gcsFileSysProperty" for key "FileSystemPr"


I don't see any value added in parameter file for file system property

AnkitCLI · 2024-12-17T06:15:23Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Close the preview
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Enter runtime argument value "projectId" for key "gcsProjectId"


Remove the properties from the macro which are already covered in the scenarios . for eg-projectID is already covered.

AnkitCLI · 2024-12-17T06:18:08Z

src/e2e-test/resources/errorMessage.properties

@@ -33,4 +33,5 @@ errorMessageMultipleFileWithoutClearDefaultSchema=Found a row with 4 fields when
 errorMessageInvalidSourcePath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidDestPath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidEncryptionKey=CryptoKeyName.parse: formattedString not in valid format: Parameter "abc@" must be
+errorMessageInvalidBucketNameSink=Spark program 'phase-1' failed with error: Errors were encountered during validation. Error code: 400, Unable to read or access GCS bucket. Bucket names must be at least 3 characters in length, got 2: 'gg'. Please check the system logs for more details.


Add only relevant error message.

itsmekumari · 2024-12-17T06:19:32Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required
+  Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true


This scenario should be from GCS source to GCS sink right? Re-check and change accordingly. And why are we making it a macro scenarios, it is already covered in macro enabled scenario anyways.

itsmekumari · 2024-12-17T06:20:06Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+  @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required
+  Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery


Use the latest existing steps from framework. Change in all the scenarios

itsmekumari · 2024-12-17T06:20:44Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Verify data is transferred to target GCS bucket


Add the validation steps in all the scenarios.

itsmekumari · 2024-12-17T06:21:52Z

src/e2e-test/features/gcs/sink/GCSSinkError.feature

@@ -65,3 +65,37 @@ Feature: GCS sink - Verify GCS Sink plugin error scenarios
    Then Select GCS property format "csv"
    Then Click on the Validate button
    Then Verify that the Plugin Property: "format" is displaying an in-line error message: "errorMessageInvalidFormat"
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST


Change the tag order, for ease of understanding.

AnkitCLI · 2024-12-17T06:57:59Z

src/e2e-test/features/gcs/sink/GCSSinkError.feature

+    Then Enter runtime argument value "gcsInvalidBucketNameSink" for key "gcsSinkPath"
+    Then Run the Pipeline in Runtime with runtime arguments
+    Then Wait till pipeline is in running state
+    Then Verify the pipeline status is "Failed"


Add open and capture logs step

itsmekumari · 2025-01-02T09:21:07Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+    Then Validate the values of records transferred to GCS bucket is equal to the values from source BigQuery table
+
+  @GCS_CSV @GCS_SINK_TEST @GCS_Source_Required @ITN_TEST


remove ITN_TEST tag

itsmekumari · 2025-01-02T09:28:03Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    When Select plugin: "GCS" from the plugins list as: "Sink"
+    Then Connect plugins: "GCS" and "GCS2" to establish connection
+    Then Navigate to the properties page of plugin: "GCS"
+    Then Select dropdown plugin property: "select-schema-actions-dropdown" with option value: "clear"


Why are we using this step?

This step is just clearing the 'output schema' clear.

I mean why are we adding this step? It is not required right?

itsmekumari · 2025-01-02T09:33:44Z

src/e2e-test/features/gcs/sink/BigQueryToGCSSink_WithMacro.feature

+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Verify data is transferred to target GCS bucket
+    Then Validate the cmek key "cmekGCS" of target GCS bucket if cmek is enabled


Add the validation step for validating the values.

In progress.

itsmekumari · 2025-01-02T09:34:03Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+      | csv        | text/csv    |
+      | tsv        | text/plain  |
+
+


remove extra line here

itsmekumari · 2025-01-06T08:33:50Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Examples:
+      | FileFormat |
+      | csv        |
+      #| tsv        |


Why are these commented , Uncomment tsv and delimited

itsmekumari reviewed Dec 17, 2024

View reviewed changes

AnkitCLI reviewed Dec 17, 2024

View reviewed changes

itsmekumari reviewed Dec 17, 2024

View reviewed changes

AnkitCLI reviewed Dec 17, 2024

View reviewed changes

AnkitCLI added the build Trigger unit test build label Dec 17, 2024

itsmekumari reviewed Jan 2, 2025

View reviewed changes

itsmekumari reviewed Jan 6, 2025

View reviewed changes

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch 5 times, most recently from 0d86052 to caf0a54 Compare January 13, 2025 04:54

All review comments are done.

9c57d24

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch from caf0a54 to 9c57d24 Compare January 14, 2025 07:52

AnkitCLI requested a review from itsankit-google January 16, 2025 05:48

E2E GCS Sink additional test scenarios. #1478

Are you sure you want to change the base?

E2E GCS Sink additional test scenarios. #1478

Conversation

bijay27bit commented Dec 17, 2024

google-cla bot commented Dec 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Jan 6, 2025 •

edited

Loading