You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Snapshot operation, when called locally, will produce one output file to the path provided by the user. Currently, the distributed version of Snapshot produces one output file per task, with the result of running the computation graph on the event range processed by that task. This difference in behaviour was introduced for performance reasons, but for the end users it can be sometimes confusing. We should introduce new behaviours to the distributed RDataFrame function:
Allow the user to request one final merged output file, understanding that this will incur in performance costs.
Allow the user to get back the list of partial output files, so they can harvest them according to further workflow requirements. Also, we should check that it's possible to write the files to a remote storage location where the user has write access (e.g. via an xrootd path).
Alternatives considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
After discussion with CMS users, it has come to my attention that in certain computing sites xrootd is not even available, in which case davix is used instead. While we know thanks to user experience that it is possible to write remote files via xrootd (provided having write permission on the remote storage location), we have no tests of writing remote data via davix (i.e. https via the TFile) with Snapshot. So we should gain experience with this case too.
Feature description
The Snapshot operation, when called locally, will produce one output file to the path provided by the user. Currently, the distributed version of Snapshot produces one output file per task, with the result of running the computation graph on the event range processed by that task. This difference in behaviour was introduced for performance reasons, but for the end users it can be sometimes confusing. We should introduce new behaviours to the distributed RDataFrame function:
Alternatives considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: