Wrangler plugin e2e tests. #650

priyabhatnagar25 · 2023-07-18T07:08:04Z

Wrangler plugin e2e tests.

vanathi-g

Seems like the build is failing because the workflow expects a plugin json file to be available but wrangler-core is not the module that generates jar + json files. wrangler-transform is the module with the actual wrangler plugin code.

vanathi-g · 2023-07-31T06:36:48Z

Try removing wrangler-core from the workflow module config, wrangler-transform has a dependency on wrangler-core anyways

vanathi-g · 2023-07-31T09:48:48Z

wrangler-transform/src/e2e-test/features/Wrangler/RunTime.feature

+  @BQ_SOURCE_TEST @BQ_SINK_TEST
+  Scenario: To verify User is able to run a pipeline using the fill null and send to error directives in the wrangler plugin
+    Given Open Datafusion Project to configure pipeline
+    Then Click on the Plus Green Button to import the pipelines


Unable to run the tests locally as it says this step (and a few others) are missing. Am I missing some config?

A PR has been raised for the steps in cdap-e2e-framework that are required for wrangler test scenarios. (cdapio/cdap-e2e-tests#193).

https://docs.google.com/document/d/11SM1qMeIA7kxrxlKi8XYdtMIjyHjnXbfp2Jr3s8fFTE/edit?resourcekey=0-zcBvSuBPBPpg4ko1tiEyJQ
I have prepared a document for wrangler. Please refer to this one.

vanathi-g · 2023-08-01T07:06:58Z

pom.xml

+          <dependency>
+            <groupId>com.google.guava</groupId>
+            <artifactId>guava</artifactId>
+            <version>31.0.1-jre</version>


edit the guava.version variable and use here

Using '${guava.version}'

update the value of guava.version to 31.0.1-jre

@vanathi-g should we update the global variable of guava version at L#103

wrangler-transform/src/e2e-test/features/Wrangler/RunTime.feature

pom.xml

vanathi-g · 2023-08-01T09:52:37Z

pom.xml

+          <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-failsafe-plugin</artifactId>
+            <version>3.0.0-M5</version>


This version is causing problems while trying to build locally. Verify once and refer to the config in database-plugins repo:
https://github.com/data-integrations/database-plugins/blob/develop/pom.xml#L634

<version>3.0.0</version> <dependencies> <dependency> <groupId>org.apache.maven.surefire</groupId> <artifactId>surefire-junit47</artifactId> <version>3.0.0</version> </dependency> </dependencies>

I upgraded the versions, but the build is still failing due to an error in generating the cucumber report file (None report file was added!). I'm looking into and debugging it.

pom.xml

poojantcs · 2023-08-10T11:14:57Z

Github actions build is currently failing as these tests require new steps in the cdap-e2e-tests repository. Once these new steps are merged in the framework repository then the Wrangler tests would be executed again.

PR link for the cdap-e2e-tests: cdapio/cdap-e2e-tests#193

saimukkamala · 2023-08-11T07:51:05Z

wrangler-transform/src/e2e-test/java/io/cdap/plugin/wrangler/actions/ValidationHelper.java

+    InterruptedException, URISyntaxException {
+    Map<String, JsonObject> bigQueryMap = new HashMap<>();
+    Map<String, JsonObject> fileMap = new HashMap<>();
+    Path importexpectedfile = Paths.get(ValidationHelper.class.getResource("/" + fileName).toURI());


can we use camel case? importExpectedFile

Renamed it to --> 'importExpectedFile'

vanathi-g · 2023-08-22T04:22:35Z

wrangler-transform/src/e2e-test/java/io/cdap/plugin/wrangler/actions/ValidationHelper.java

+    getBigQueryTableData(table, bigQueryMap);
+    getFileData(importExpectedFile.toString(), fileMap);
+
+    boolean isMatched = matchJsonMaps(bigQueryMap, fileMap);


Why not just use bigqueryMap.equals(fileMap) and remove the matchJsonMaps function? The map equals method also functionally does the same thing (compare keys and corresponding values)

done removed that function

vanathi-g

LGTM, please squash commits and merge.

The RunTime.feature class may have to be renamed in the next PR depending on what additional directive test cases are being added.

vanathi-g · 2023-08-22T06:53:06Z

wrangler-transform/src/e2e-test/resources/BQtesdata/BigQuery/BigQueryInsertDataQuery.txt

@@ -0,0 +1,9 @@
+insert into `DATASET.TABLE_NAME` (ID, name, email, Customer_id, First_name, Last_name, Age, Address, Col1,


Since the "body" column is used in the second test (parse csv, fill null, send to error) update the data to reflect an actual csv string.

Also, update the address column to have one null value to ensure the send-to-error if empty directive that is used is working correctly.

vanathi-g · 2023-08-22T07:14:50Z

...ta/Wrangler/Wrangler_Directive_Copy_Count_DeletecolumnTestPipeline-1-cdap-data-pipeline.json

+                    "properties": {
+                        "field": "*",
+                        "precondition": "false",
+                        "directives": "parse-as-csv :body ',' true\ncopy :Customer_id :Customer_id_copy true\ndrop :Address\nset-column :name_count string:length(name)",


In the directive:
parse-as-csv :body ',' true

the 'true' argument is used to take the first value of the column as the header. There is actually no need to use this directive in this test. The current test has the copy directive's force argument set to true, so change the destination column to an existing column and validate the result accordingly.

Fix the usage in the other files as well to account for the first row being dropped in the results.

vanathi-g · 2023-08-22T09:51:42Z

I would suggest adding just one .feature file per PR which tests atleast 7-8 directives in the same pipeline. For example, the directives can be grouped as such -

parser - 1 (ensure the actual data is in the format being parsed)
column operations - 4
row operations - 3

Can refer to this to get the groupings: https://cdap.atlassian.net/wiki/spaces/DOCS/pages/380600349/Directives

Change this PR to have just one scenario to begin with and ensure the data in BQ also reflects the behaviour we want to test, thanks!

AnkitCLI · 2023-09-06T06:07:42Z

closing pull request as new PR is raised for Wrangler directives.

priyabhatnagar25 force-pushed the Import_e2e_Wrangler_plugin branch from 9fdb967 to 43c1334 Compare July 18, 2023 07:11

Vipinofficial11 requested a review from saimukkamala July 21, 2023 10:34

Vipinofficial11 added the build Triggers unit test build label Jul 25, 2023

sau42shri requested a review from vanathi-g July 26, 2023 06:14

Vipinofficial11 added build Triggers unit test build and removed build Triggers unit test build labels Jul 26, 2023

Vipinofficial11 assigned priyabhatnagar25 and bharatgulati Jul 28, 2023

vanathi-g requested changes Jul 31, 2023

View reviewed changes

vanathi-g reviewed Jul 31, 2023

View reviewed changes

vanathi-g requested changes Aug 1, 2023

View reviewed changes

vanathi-g reviewed Aug 1, 2023

View reviewed changes

pom.xml Show resolved Hide resolved

saimukkamala approved these changes Aug 11, 2023

View reviewed changes

AnkitCLI force-pushed the Import_e2e_Wrangler_plugin branch 2 times, most recently from 32735a4 to d98da22 Compare August 21, 2023 09:13

vanathi-g reviewed Aug 22, 2023

View reviewed changes

vanathi-g approved these changes Aug 22, 2023

View reviewed changes

Wrangler plugin e2e tests

5bbdd61

AnkitCLI force-pushed the Import_e2e_Wrangler_plugin branch from 6984bcb to 5bbdd61 Compare August 22, 2023 05:57

vanathi-g requested changes Aug 22, 2023

View reviewed changes

vanathi-g reviewed Aug 22, 2023

View reviewed changes

AnkitCLI and others added 2 commits August 22, 2023 15:39

Addressed comments

bb39338

Merge branch 'data-integrations:develop' into Import_e2e_Wrangler_plugin

6f36fae

bharatgulati closed this Sep 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrangler plugin e2e tests. #650

Wrangler plugin e2e tests. #650

priyabhatnagar25 commented Jul 18, 2023

vanathi-g left a comment

vanathi-g commented Jul 31, 2023

vanathi-g Jul 31, 2023

bharatgulati Jul 31, 2023

priyabhatnagar25 Jul 31, 2023

vanathi-g Aug 1, 2023

bharatgulati Aug 1, 2023

vanathi-g Aug 17, 2023

AnkitCLI Aug 17, 2023

vanathi-g Aug 1, 2023

bharatgulati Aug 1, 2023

poojantcs commented Aug 10, 2023

saimukkamala Aug 11, 2023

bharatgulati Aug 11, 2023

vanathi-g Aug 22, 2023

AnkitCLI Aug 22, 2023

vanathi-g left a comment

vanathi-g Aug 22, 2023

rahuldash171 Aug 28, 2023

vanathi-g Aug 22, 2023

vanathi-g commented Aug 22, 2023

AnkitCLI commented Sep 6, 2023

		@@ -0,0 +1,9 @@
		insert into `DATASET.TABLE_NAME` (ID, name, email, Customer_id, First_name, Last_name, Age, Address, Col1,

Wrangler plugin e2e tests. #650

Wrangler plugin e2e tests. #650

Conversation

priyabhatnagar25 commented Jul 18, 2023

vanathi-g left a comment

Choose a reason for hiding this comment

vanathi-g commented Jul 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

poojantcs commented Aug 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vanathi-g left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vanathi-g commented Aug 22, 2023

AnkitCLI commented Sep 6, 2023