Skip to content

Commit

Permalink
fix:Update README.md (#107)
Browse files Browse the repository at this point in the history
* fix:Update README.md
  • Loading branch information
patduin committed Aug 29, 2024
1 parent 340ea3f commit f3b532f
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion hive-event-listeners/apiary-gluesync-listener/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The GlueSync listener can be configured by setting the following System Environm
GLUE_PREFIX|No|Prefix added to Glue databases to handle database name collisions when synchronizing multiple metastores to the Glue catalog.

## Table update SkipArchive
[AWS default](https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateTable.html#Glue-UpdateTable-request-SkipArchive) is to archive the table on every update. This especially with Iceberg tables can lead to a lot of table version of which you can only have a certain limit. To counter this we override this property and set skipArchive=true so do *not* make an archive of the table when updating.
[AWS default](https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateTable.html#Glue-UpdateTable-request-SkipArchive) is to archive the table on every update. With Iceberg tables this can lead to a lot of table versions. In Glue you can only have a certain limit of the number of versions and you'll get exceptions when trying to update a table once you hit that limit. Manual version removal through AWS api is then needed. To counter this we override this property and set skipArchive=true. So the listners does *not* make an archive of the table when updating.
If an archive is needed, this can be done per table by setting the Hive table property: 'apiary.gluesync.skipArchive=false'.


Expand Down

0 comments on commit f3b532f

Please sign in to comment.