Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Scio to 0.14.12; run CI on Java 17; use magnolify-bigquery in BigDiffy #784

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

clairemcginty
Copy link
Contributor

  • Upgrades Scio to 0.14.12, which unblocks Java 17+
  • Updates CI to run on Java 17 (thus the added JavaOptions.scala file, which adds options required for 17+). Note that the project still has the -target 8 scalaOpt so it will continue to produce Java8-compatible bytecode
  • Replaces BigQueryType usage in BigDiffy with magnolify-bigquery. BigQueryType is deprecated and less performant than Magnolify for BQ writes

Testing: Packaged locally and tested BigDiffy with BQ output to validate that all three tables (_fields, _global, and _keys) are produced correctly.

Copy link

codecov bot commented Feb 10, 2025

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 70.68%. Comparing base (b4e0ddd) to head (f4ec64c).

Files with missing lines Patch % Lines
...in/scala/com/spotify/ratatool/diffy/BigDiffy.scala 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #784      +/-   ##
==========================================
+ Coverage   70.61%   70.68%   +0.06%     
==========================================
  Files          44       44              
  Lines        1865     1859       -6     
  Branches      309      315       +6     
==========================================
- Hits         1317     1314       -3     
+ Misses        548      545       -3     
Flag Coverage Δ
ratatoolCli 2.89% <0.00%> (+0.01%) ⬆️
ratatoolCommon 0.00% <ø> (ø)
ratatoolDiffy 32.29% <0.00%> (+0.12%) ⬆️
ratatoolExamples 17.20% <0.00%> (+0.05%) ⬆️
ratatoolSampling 60.53% <ø> (-0.11%) ⬇️
ratatoolScalacheck 77.94% <ø> (ø)
ratatoolShapeless 4.16% <0.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@benkonz
Copy link
Contributor

benkonz commented Feb 11, 2025

thanks @clairemcginty! I tested your changes with BigSampler and BigDiffy. Both succeeded, and the changes look good to me. TY!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants