Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically add Athena partition when new test results are available #50

Open
binarylogic opened this issue Apr 3, 2020 · 0 comments

Comments

@binarylogic
Copy link
Contributor

In order for Athena to query new benchmarking data is must know about the partition. This is achieved by running a repair query:

MSCK REPAIR TABLE vector_tests

This query is expensive and takes a few minutes to execute. This is because it's listing every file in the bucket in order to discover new partitions. A much more efficient method is to add the partition directly:

ALTER TABLE vector_tests ADD PARTITION (...) location 's3://test-results.vector.dev/../../../'

For example, this could be done automatically by listening for new S3 files being placed in the bucket and firing a lambda to run this query.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant