-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
optimize disk cache io #21876
base: master
Are you sure you want to change the base?
optimize disk cache io #21876
Conversation
@tjgq Have there been any experiments at Google on replacing some of the custom filesystem functions with Java NIO? I recently used |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am sympathetic to the objective of this PR (the sendfile/fcopyfile optimization in NIO definitely sounds worthwhile), but I'd like to do some benchmarking of my own before we accept it. That will probably take a while, as I'm on vacation next week.
// FIXME: Do we need it? | ||
// try (FileOutputStream out = new FileOutputStream(outFile)) { | ||
// // Fsync temp before we rename it to avoid data loss in the case of machine | ||
// // crashes (the OS may reorder the writes and the rename). | ||
// out.getFD().sync(); | ||
// } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need it unless you can convince me that the scenario described in the comment is not a concern (https://danluu.com/deconstruct-files/ is a good reference on what can go wrong when a system crashes in the middle of a filesystem operation).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I looked at the NIO source code. Only public static Path copy(Path source, Path target, CopyOption... options) throws IOException
can optimize the IO operation, and there is no fsync method available.
Utilize NIO instead of InputStream. For MacOS, NIO copy can employ fcopyfile for non-blocking IO copy to optimize IO performance.