-
Notifications
You must be signed in to change notification settings - Fork 0
/
future.tex
41 lines (34 loc) · 3.4 KB
/
future.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
\section{Future Work}
\label{sec:conclusion}
This paper covered the design and implementation of the \connector{} which
collected I/O data from the Darshan I/O characterization tool to create new time series data sets
that enable further insights into I/O behavior and patterns. Five key components were used
to develop this design: the application I/O event data collector (Darshan), lightweight data
transport (LDMS Streams), efficient storage (DSOS), analysis (Python modules), and
visualization (Grafana). Results of this design add enhancements to both LDMS and Darshan
tools as well as create new insights and provide a better understanding of application
I/O performance and behavior.
Our next steps are to further expand the \connector{} and it's capabilities by
including more I/O event data and demonstrating advanced insights into correlations
between I/O performance and system behavior and providing the
capability for overhead reduction through sampling and/or aggregation techniques that still
provide enough resolution for a user to gain run time insight into the I/O behavioral
characteristics of an application and to correlate these characteristics with those of
related system components. We will also be performing more overhead analysis over a variety
of I/O intensive applications.
%However, an issue with increased overhead does arise when collecting data
%from short but intensive I/O applications (average 500,000 msgs/sec). We performed additional experiments without the json message formatting (e.g. did not publish collected data to \emph{LDMS} streams interface) and we observed an 80-90\% increase in total runtime from the \connector{}.
%The \connector{} is implemented such that when Darshan detects an I/O event, the \connector{} will collect and format that current set of I/O metrics into a json message. In order to send a json message, all integers must be converted to strings and this conversion comes at a performance cost. Since there is currently no other way to send I/O data as a json message to the LDMS Streams interface without converting the integers to strings, we must pay a performance cost.
%This indicates that the json message formatting might also be a factor in increased overhead.
%To address this issue, we will include an option for users to decide the rate of I/O events that the \connector{} will collect and format into a json message. Having this option will allow users who are running intensive I/O applications to still be able to analysis runtime time series data of their application without concern of the runtime performance.
The \Darshan{} will be made available as an optional "module" plugin to the Darshan tool so
Darshan users can collect time series data without increasing memory impact on compute nodes
and better understand applications I/O performance across HPC systems and clusters.
\section{Acknowledgment}
The authors would like to thank Jim Brandt (SNL) and Devesh Tiwari (Northeastern), for useful discussions and suggestions in this work and Darshan contributors, Phil Carns (ANL) and Shane Snider (ANL), for insights about Darshan architecture.
%\RED{
%\begin{itemize}
% \item Explain the purpose of the future work.
% \item What are our plans for this connector? Will we be implementing \emph{LDMS Streams} across other I/O characterization tools, expanding the streams capabilities on Darshan, testing across other applications, etc.?
%\end{itemize}
%}