2020-05-08 (GCS 2.1.3, BQ 1.1.3)

Changelog

Cloud Storage connector:

1. Add support for Cloud Storage objects CSEK encryption:

   fs.gs.encryption.algorithm (not set by default)
   fs.gs.encryption.key (not set by default)
   fs.gs.encryption.key.hash (not set by default)
   

2. Update all dependencies to latest versions.

3. Added a new output stream type which can be used by setting:

   fs.gs.outputstream.type=FLUSHABLE_COMPOSITE
   

   The FLUSHABLE_COMPOSITE output stream type behaves similarly to the
   SYNCABLE_COMPOSITE type, except it also supports hflush(), which uses
   the same implementation with hsync() in the SYNCABLE_COMPOSITE output
   stream type.

4. Added a new output stream parameter

   fs.gs.outputstream.sync.min.interval.ms (default: 0)
   

   to configure the minimum time interval (milliseconds) between consecutive
   syncs. This is to avoid getting rate limited by GCS. Default is 0 - no
   wait between syncs. hsync() when rate limited will block on waiting for
   the permits, but hflush() will simply perform nothing and return.

5. Restore compatibility with pre-2.8 Hadoop versions.

Big Query connector:

1. Update all dependencies to latest versions.