Skip to content

Latest commit

 

History

History
10 lines (6 loc) · 828 Bytes

spark-service-outputcommitcoordinator.adoc

File metadata and controls

10 lines (6 loc) · 828 Bytes

OutputCommitCoordinator

From the scaladoc (it’s a private[spark] class so no way to find it outside the code):

Authority that decides whether tasks can commit output to HDFS. Uses a "first committer wins" policy. OutputCommitCoordinator is instantiated in both the drivers and executors. On executors, it is configured with a reference to the driver’s OutputCommitCoordinatorEndpoint, so requests to commit output will be forwarded to the driver’s OutputCommitCoordinator.

The most interesting piece is in…​

This class was introduced in SPARK-4879; see that JIRA issue (and the associated pull requests) for an extensive design discussion.