Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds helper methods for record insertion to BQ table. #69

Closed
wants to merge 44 commits into from
Closed

Adds helper methods for record insertion to BQ table. #69

wants to merge 44 commits into from

Conversation

prashastia
Copy link
Collaborator

Contains different methods

  1. That allows generation of records and storing them to avro files
  2. Transferring the avro files to BQ tables
  3. Deleting the thus generated avro files.

The records are generated according the specified table type.

Separate write record implementations for different types of tables.

@prashastia
Adds a new module for nightly tests.
a4d0d25 
This module is similar to the BigQueryExample. A few changes to count the number of records and log them.
@prashastia
Modifies the docstring.
f674638
@prashastia
Modifies the docstring.
6ec22dc
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
f0cd825 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
c814bd9 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
f5d3f31 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
b9cad97 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
b3ce92b 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Adds simple e2e test, Adds parse_logs.py, Adds table_read.sh
4e6ef67 
This test reads a simpleTable.
Shell script and python script to check the number of records read.
@prashastia
Modifies IntegrationTest to check query correctness.
dd8e5be
@prashastia
Adds spotless:apply.
137030d 
comments CODECOV_TOKEN usage.
@prashastia
Renames the table_read file to bounded_table_read.sh
957ebf0
@prashastia
Addresses review comments.
2ff4032
@prashastia
Creates separate shell script for bounded jobs.
394132d
@prashastia
Adds a new common shell script for common actions performed for both …
5425806 
…bounded and unbounded source.
@prashastia
Addresses review comments.
ea79601
@prashastia
Addresses review comments.
c6a564f
@prashastia
Formats the file.
a2a49b3
@prashastia
Fixes checkstyle violations.
3da1b14
@prashastia
Fixes checkstyle violations.
0f34c03
@prashastia
Fixes checkstyle violations.
0822681
@prashastia
Fixes checkstyle violations.
1770526
@prashastia
Changes test name, adds a new e2e test for checking table read for ta…
e416b1f 
…ble with complex schema.
@prashastia
Changes test name, adds a new e2e test for checking table read for ta…
471aca2 
…ble with complex schema.
@prashastia
Adds a new e2e test for checking query read.
68d48d0
@prashastia
Fixes cause of error in query read execution.
44f39f6
@prashastia
Fixes cause of error in query read execution.
b8a5883
@prashastia
Adds a new e2e test for checking large table ~200GBs read.
46ac1e6
@prashastia
Adds utils.py - a class containing implementations for writing recor…
74afc54 
…ds to different tables required for the e2e tests.
@prashastia
Adds table_type.py - a class containing various methods for record g…
41a0639 
…eneration.

 The records can be string, long, records type.
@prashastia
Copy link
Collaborator Author

/gcbrun

@prashastia
insert_dynamic_partitions.py A python script to insert partitions inc…
92835ec 
…rementally into the partitioned table. Used by the e2e unbounded source testing script.
@prashastia
Adds shell script that creates the asynchronously running unbounded j…
2925f6c 
…ob, Dynamically adds partitions and then kills the dataproc job.
@prashastia
Adds a new step in hte nightly job - the unbounded table read. Also a…
535905f 
…dds the call to unbounded_table_read.sh via the common table_read.sh
@prashastia
Modifies yaml file to add unbounded table read step to be run during …
01031a5 
…the e2e tests.
@prashastia
Attempts to fix the execution of unbounded table read.
41a48e0
@prashastia
Attempts to fix the execution of unbounded table read.
f1b6758
@prashastia
Attempts to fix the execution of unbounded table read.
6cf11e3
@prashastia
Removes the extra sleep afterwards, added in shell instead.
6300733
@prashastia
Modifies utils.py to add an extra class responsible for argument inpu…
da31fc8 
…t and validation.
@prashastia
Modifies parse_logs.py to use the common utils method for argument in…
8b0bcc9 
…put.
@prashastia
Modifies insert_dynamic_partitions.py to use the common utils method …
da20ee4 
…for argument input.
@prashastia
Adds a python script to create a partitioned table and add initial va…
33a5e83 
…lue to it.
@prashastia
Modifies unbounded_table_read.sh to delete preexisting partitioned ta…
7793004 
…ble and create new one.
@prashastia
undo.
bc2d875
@prashastia prashastia closed this Dec 23, 2023
@prashastia prashastia deleted the nightly-tests-unbounded-read-1 branch December 23, 2023 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jayehwhyehentee jayehwhyehentee Awaiting requested review from jayehwhyehentee

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@prashastia