From 0415b5c9e9e0be906ce17446834f2aadc2655b79 Mon Sep 17 00:00:00 2001 From: Monthon Klongklaew Date: Thu, 10 Oct 2024 18:28:26 +0100 Subject: [PATCH] Update documentation related to the prefetcher (#1049) * Update documentation related to the prefetcher Update changelog to include changes in the prefetcher and document the unstable configuration to set maximum prefetch window size. Signed-off-by: Monthon Klongklaew * PR comments Signed-off-by: Monthon Klongklaew * Change wording Signed-off-by: Monthon Klongklaew * Apply PR suggestion Co-authored-by: Daniel Carl Jones Signed-off-by: Monthon Klongklaew --------- Signed-off-by: Monthon Klongklaew Co-authored-by: Daniel Carl Jones --- doc/CONFIGURATION.md | 13 +++++++++++++ mountpoint-s3/CHANGELOG.md | 4 ++++ 2 files changed, 17 insertions(+) diff --git a/doc/CONFIGURATION.md b/doc/CONFIGURATION.md index ff7fcf2ed..5c2b876a2 100644 --- a/doc/CONFIGURATION.md +++ b/doc/CONFIGURATION.md @@ -358,3 +358,16 @@ We welcome feedback on how this works for your applications and workloads. ## Logging By default, Mountpoint emits high-severity log information to [syslog](https://datatracker.ietf.org/doc/html/rfc5424) if available on your system. You can change what level of information is logged, and to where it is logged. See [LOGGING.md](LOGGING.md) for more details on configuring logging. + +## Unstable configurations + +Configurations in this section are experimental and may be removed or superseded by other configurations in any future release. + +### Maximum prefetch window size + +Mountpoint downloads object contents into memory ahead of read requests in order to optimize sequential-read throughput. The prefetch window size determines how much data Mountpoint would prefetch data into memory and the value can be different for each file handle. These window sizes are adjusted automatically based on available system memory, scaling up to a maximum of 2 GiB per file handle by default. + +This should work well for most use cases but if you need to use a different value, you can set the maximum prefetch window size to an arbitrary number with the `UNSTABLE_MOUNTPOINT_MAX_PREFETCH_WINDOW_SIZE` environment variable at mount time, providing a value in bytes. + +> [!WARNING] +> Overriding the default maximum prefetch window size may reduce overall read throughput. diff --git a/mountpoint-s3/CHANGELOG.md b/mountpoint-s3/CHANGELOG.md index e9de5d164..bb0e42d19 100644 --- a/mountpoint-s3/CHANGELOG.md +++ b/mountpoint-s3/CHANGELOG.md @@ -1,5 +1,9 @@ ## Unreleased +### New features + +* Mountpoint now automatically adjusts its prefetcher read window size based on available system memory. This reduces the risk of Mountpoint potentially consuming all available system memory in cases where a large number of file handles are read from concurrently. ([#1013](https://github.com/awslabs/mountpoint-s3/pull/1013)) + ### Breaking changes * When configured to log to a directory, Mountpoint now includes a random string following the timestamp in the file name.