You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
From reading the code and looking at behavior of when using the SDK for S3, I believe AsyncResponseTransformer.toBlockingStream() produces a stream that will store the all the data in memory if not consumed fast enough. This is a problem when downloading lots of data from S3 without being able to write it fast enough, and it causes out of memory issues. I believe there should be a limit on how much ByteBufferStoringSubscriber will accumulate.
Expected Behavior
When reading faster than consuming with an input stream, I don't expect the input stream to buffer the entire remaining amount.
Current Behavior
Input streams can buffer the full amount of data if not written fast enough.
Reproduction Steps
I haven't tried this, but if my theory is right:
Attempt to download a download a large object with S3 using blocking input stream
Don't consume the input stream.
The full data should reside in memory.
Possible Solution
ByteBufferStoringSubscriber or it's delegate should set a max number of bytes to store along with a minimum.
Additional Information/Context
It's possible that I'm getting out of memory errors from downloading too much data from S3 for a different reason, but this is what my debugging has lead me to.
AWS Java SDK version used
2.24.1
JDK version used
21
Operating System and version
Ubuntu 22.04
The text was updated successfully, but these errors were encountered:
I might be wrong about this bug report because ByteBufferStoringSubscriber does try to limit to the buffer. However, if the a single event exceeds the 4MB buffer there can be more than 4MB stored.
I think this is more a problem that the crt client sets the initial read buffer high (80mb) by default, so lots of medium size files can quickly get put into memory. Closing this as my memory issues when away after setting that much lower. I think the default should probably much lower than 80mb.
This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
Describe the bug
From reading the code and looking at behavior of when using the SDK for S3, I believe AsyncResponseTransformer.toBlockingStream() produces a stream that will store the all the data in memory if not consumed fast enough. This is a problem when downloading lots of data from S3 without being able to write it fast enough, and it causes out of memory issues. I believe there should be a limit on how much ByteBufferStoringSubscriber will accumulate.
Expected Behavior
When reading faster than consuming with an input stream, I don't expect the input stream to buffer the entire remaining amount.
Current Behavior
Input streams can buffer the full amount of data if not written fast enough.
Reproduction Steps
I haven't tried this, but if my theory is right:
Possible Solution
ByteBufferStoringSubscriber or it's delegate should set a max number of bytes to store along with a minimum.
Additional Information/Context
It's possible that I'm getting out of memory errors from downloading too much data from S3 for a different reason, but this is what my debugging has lead me to.
AWS Java SDK version used
2.24.1
JDK version used
21
Operating System and version
Ubuntu 22.04
The text was updated successfully, but these errors were encountered: