Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add AWSS3 as a package extension #147

Merged
merged 7 commits into from
Nov 16, 2023
Merged

add AWSS3 as a package extension #147

merged 7 commits into from
Nov 16, 2023

Conversation

ericphanson
Copy link
Member

@ericphanson ericphanson commented Nov 16, 2023

To automatically use the efficient byte-range method when AWSS3 is loaded, on Julia 1.9+. This is currently pirated in various places, which should be removed on 1.9+.

I have used the strategy from here such that the feature is only present on 1.9+, but there's no dependency cost pre-1.9. An alternative would be to add AWSS3 as a full dependency pre-1.9.

@ericphanson
Copy link
Member Author

Hm, codecov says the read_byte_range method I added isn't getting hit 🤔

Copy link
Member

@kleinschmidt kleinschmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great! another thing we've talked about internally in teh context of using a package extension here is supporting atomically capturing the versionId generated on write. we can't do that without having methods for Onda.store(::S3Path, ...) I think.

that might be more complex than we want to tackle in this initial PR though so I'm happy to let it go for now!

test/awss3.jl Outdated
Comment on lines 42 to 45
# Load subspan to exercise method
span = TimeSpan(0, Second(1))
loaded_span = Onda.load(signal, span; encoded=true)
@test loaded_samples[:, span] == loaded_span
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for some reason it doesn't seem like this has hit that added method, at least according to the "missed coverage" annotations above. I wonder if somehow the package extension isn't getting loaded properly in tests?

we also have some tests in OndaBatches.jl which try to really make sure we're actually passing the byte range to AWSS3: https://github.com/beacon-biosignals/OndaBatches.jl/blob/a43b6f0a6837c2a478a1845c69728c1851e234c0/test/utils.jl#L22-L27

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

locally I confirmed it was loading, but I agree something seems off here. Weirdly codecov says the other method WAS hit in the extension code...

test/awss3.jl Outdated Show resolved Hide resolved
ericphanson and others added 2 commits November 16, 2023 16:48
Co-authored-by: Dave Kleinschmidt <dave.f.kleinschmidt@gmail.com>
test/awss3.jl Outdated Show resolved Hide resolved
@ericphanson
Copy link
Member Author

Turns out the issue was sloppily copying the code to create the signal, since I was using compressed LPCM which of course we don't do byte range requests for currently

test/awss3.jl Outdated Show resolved Hide resolved
@ericphanson
Copy link
Member Author

Remaining coverage loss is we aren't hitting the method ambiguity method - do you happen to remember how to trigger this @kleinschmidt ?

@kleinschmidt
Copy link
Member

Remaining coverage loss is we aren't hitting the method ambiguity method - do you happen to remember how to trigger this @kleinschmidt ?

hmm, I don't off teh top of my head. but it does seem to get hit in OndaBatches.jl tests; https://app.codecov.io/gh/beacon-biosignals/OndaBatches.jl/blob/main/src%2Futils.jl#L18

@kleinschmidt
Copy link
Member

I think that path gets hit when you load without a span...

@ericphanson
Copy link
Member Author

I think that path gets hit when you load without a span...

We already do try that, but I suspect maybe that happens in the compressed case, so going to try both compressed & uncompressed like in the OndaBatches tests

Copy link
Member

@kleinschmidt kleinschmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few little formatting thing (no auto-formatter apparently) but

test/awss3.jl Outdated Show resolved Hide resolved
test/awss3.jl Outdated Show resolved Hide resolved
Co-authored-by: Dave Kleinschmidt <dave.f.kleinschmidt@gmail.com>
@ericphanson ericphanson merged commit d8735c8 into main Nov 16, 2023
5 checks passed
@ericphanson ericphanson deleted the eph/extension branch November 16, 2023 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants