Adds a "read from query results" example #53

prodriguezdefino · 2023-07-04T20:55:06Z

No description provided.

…ed table is expired and no longer accessible.

…rom time to time, the pipeline should be able to recover reading from the beginning of the split or the last checkpointed offset

...igquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java

.../src/main/java/com/google/cloud/flink/bigquery/source/split/BigQuerySourceSplitAssigner.java

...igquery-examples/src/main/java/com/google/cloud/flink/bigquery/examples/BigQueryExample.java

...bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java

...uery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java

...uery/src/main/java/com/google/cloud/flink/bigquery/table/restrictions/BigQueryPartition.java

sonatype-lift · 2023-07-04T20:58:12Z

...ry/src/main/java/com/google/cloud/flink/bigquery/common/utils/GoogleCredentialsSupplier.java

+
+    private static Credentials createCredentialsFromFile(String file) {
+        try {
+            return GoogleCredentials.fromStream(new FileInputStream(file));


PATH_TRAVERSAL_IN: This API (java/io/FileInputStream.(Ljava/lang/String;)V) reads a file whose location might be specified by user input

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

sonatype-lift · 2023-07-04T20:58:14Z

...-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/source/BigQuerySource.java

+ */
+@AutoValue
+@PublicEvolving
+public abstract class BigQuerySource<OUT>


AutoValueCouldNotWrite: Could not write generated class com.google.cloud.flink.bigquery.source.AutoValue_BigQuerySource: javax.annotation.processing.FilerException: Attempt to recreate a file for type com.google.cloud.flink.bigquery.source.AutoValue_BigQuerySource

❗❗ 3 similar findings have been found in this PR

🔎 Expand here to view all instances of this finding

File Path Line Number

flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/CredentialsOptions.java 34

flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/source/config/BigQueryReadOptions.java 41

flink-connector-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/config/BigQueryConnectOptions.java 34

Visit the Lift Web Console to find more details in your report.

ℹ️ Expand to see all @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

...tor-bigquery/src/main/java/com/google/cloud/flink/bigquery/common/utils/SchemaTransform.java

sonatype-lift · 2023-07-04T20:58:18Z

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.¹

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/flink-bigquery-connector/53.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/flink-bigquery-connector/53.diff | git apply

Once you're satisfied, commit and push your changes in your project.

You can preview the patch by opening the patch URL in the browser. ↩

…_error

…name

…ererror

jayehwhyehentee · 2023-11-28T06:30:47Z

/gcbrun

jayehwhyehentee · 2023-12-04T19:14:45Z

/gcbrun

...igquery-examples/src/main/java/com/google/cloud/flink/bigquery/examples/BigQueryExample.java

jayehwhyehentee · 2023-12-05T15:42:52Z

/gcbrun

prodriguezdefino added 18 commits June 27, 2023 13:31

cleaning up the main branch for new version.

eb9dc8f

updates in ignore and readme files

0695673

prepping the pom addition, added parent's compliance tools

e36cc44

adding parent pom and the connector impl project pom

e167803

adding common functionalities

8043378

added the bigquery services wrapper and factories

b455b4e

creates the split, its state and the enumerator state

27cd837

added configs, split reader and split assignment impls

4d4b60f

applying recommendations from sonartype-lift

0567b58

adding the Datastream source implementation for BigQuery

9006714

added Table API implementation for BigQuery

d5d95bf

adding the example and shaded distro jar, fixes a NPE when the provid…

1263768

…ed table is expired and no longer accessible.

added covertura analysis through open clover and improved few tests

4e185bd

fixes partition and row restriction orden for push downs

4e63b02

adds a test which make the storage read server stream iterator fail f…

74d5b90

…rom time to time, the pipeline should be able to recover reading from the beginning of the split or the last checkpointed offset

renamed example's packages and pom's group id to the correct one

aff1d6a

resolve locally errors that occur in the split reader

57d5a48

added a read from query results example

b16392f