-
Notifications
You must be signed in to change notification settings - Fork 0
/
approach.tex
74 lines (60 loc) · 7.59 KB
/
approach.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
\section{Darshan LDMS Integration}
\label{sec:DarshanLDMSIntegration}
%Therefore, this paper provides an existing system level system
Darshan collects I/O application data for post-run analysis, and LDMS has a low
overhead sampling capability supported by fast storage and a modern web interface.
We chose to enhance both Darshan and LDMS by adding a sampling capability that acts
as a connector between these two applications to support I/O runtime data sampling
and visualization. In this section, we present a high-level overview of the design
and implementation of the \Darshan{}. The components used to create this infrastructure are:
\begin{itemize}
\item The I/O characterization tool, Darshan, to collect application I/O behavior and patterns.
%However, this tool does not report the \emph{absolute timestamp} so modifications to the code were made to expose this data.
\item LDMS to provide and transport live run time data feed about application I/O events.~\cite{ldmsgithubwiki}
\item DSOS to store and query large amounts of data generated on a production HPC system.~\cite{sosgithub}
\item HPC Web Services to present run time I/O data to enable the user to create new meaningful analyses~\cite{ClusterAV}.
\end{itemize}
%This approach will provide run time insights about application I/O by using the following tools:
%\begin{enumerate}
% \item The I/O characterization tool, Darshan, to collect application I/O behavior and patterns. However, this tool does not report the \emph{absolute timestamp} so modifications to the code were made to expose this data.
% \item LDMS to provide and transport live run time data feed about application I/O events.~\cite{ldmsgithubwiki}
% \item DSOS to store and query large amounts of data generated on a production HPC system.~\cite{sosgithub}
% \item HPC Web Services to present run time I/O data. The timeseries data will enable the user to identify when a variability occurs as well as create new meaningful analyses.~\cite{ClusterAV}
%\end{enumerate}
%Darshan is used to tune applications for increased scientific productivity or performance and is suitable for full time deployment for workload characterization of large systems~\cite{darshan-webpage}. It provides detailed statistics about various level file accesses made by MPI and non-MPI applications which can be enabled or disabled as desired. These levels include POSIX, STDIO, LUSTRE and MDHIM for non-MPI applications and MPI-IO, HDF5 and some PnetCDF for MPI applications~\cite{darshan-runtime}. This functionality provides users with a summary of I/O behavior and patterns from an application run but it does so post-run. Therefore, it does not allow insights into \emph{run time} I/O behavior and patterns which makes it nearly impossible to identify the root cause(s) of I/O variability and when this occurs.
LDMS is used to efficiently and scalably collect and transport \emph{synchronous}
and \emph{event-based} data with low-overhead. Two key functionalities the \Darshan{}
will leverage in order to create the \connector{} are the LDMS \emph{streams}
publish-subscribe bus and LDMS transport~\cite{ldmsgithub}. In this work, we
enhanced LDMS to support application I/O data injection and store it to DSOS.
We also modified Darshan to expose the \emph{absolute timestamp} and publish runtime I/O events of each rank to \emph{LDMS Streams}. This integration will be described in
detail later on in the paper.
%LDMS is used to efficiently collect and transport scalable \emph{synchronous} and \emph{event-based} data with low-overhead. Two key functionalities it has that will be leveraged in the \Darshan and create the \connector are the \emph{LDMS Streams} and transport~\cite{ldmsgithub}. In the \Darshan, the LDMS was enhanced to support application I/O data injection and store to DSOS while Darshan was modified to expose the \emph{absolute timestamp} and publish run time I/O events for each rank to the \emph{LDMS Streams}. This integration will be described in detail later on.
%Implementing LDMS into Darshan will provide timestamped I/O event data during an application run. This data will provide any users, application developers or researchers to further their analyses, better understand when an I/O variability occurs and can identify any correlations between the I/O performance and file system, network congestion or system contention.
\emph{DSOS} enables the ability to query the timestamped application I/O data through
a variety of APIs while Grafana will provide a front-end interface for visualizing the
stored data that has been queried and analyzed using Python based modules on the back-end.
With these tools, users can view, edit and share analyses of the data as well as
create new meaningful analyses.
%The implementation of LDMS into Darshan along with the storage, analysis and visualization components that make up this design will provide detailed insights into the I/O behavior and patterns during run time. This insight will allow users and researchers to better understand how application I/O variability correlates with overall system behavior (e.g. file system, network congestion, etc.) how the time of day affects the I/O performance, how the I/O pattern within a run affects the I/O performance of read and write and why the read and write I/O performance patterns are different and independent of each other. Further, the occurrence of any I/O variability can be identified during runtime.
%\RED{\begin{itemize}
% \item Address current I/O performance analysis and implications.
% \subitem Lesson Learned: Different applications experience high and low performance variations at separate and disjoint time periods.These times periods are shared across different applications and clusters.
% \subitem How LDMS will help: It is suggested that a simple I/O monitoring data collection from Darshan will be useful in identifying these time periods. LDMS will be able to provide the run time I/O monitoring data collection which will help them in identifying these time periods.
% \subitem Lesson Learned: Clusters running on weekends observe some of the highest performance variations. It is assumed that this could be from I/O intensive application runs during the weekends.
% \subitem How LDMS will help: The I/O timeseries data generated by LDMS will provide NERSC researchers with new data they can utilize to further their research.
% \item Benefits of having time series data:
% \subitem Better understand how application I/O variability correlates with overall system behavior
% \subitem Understand how time of day affects the I/O performance
% \subitem Understand how the I/O pattern within a run affects the I/O performance of read and write. E.g. perhaps an application that has nodes writing data at disjoint times have less I/O performance variation than one that has all nodes writing data at the same time.
% \subitem Understand why read and write I/O performance patterns are different and independent of each other. E.g. read and write may exhibit different behavior during the run, which affects the performance of each in unique ways
%\end{itemize}
%}
%\subsection{Darshan and LDMS Overview}
%\RED{ \begin{itemize}
% \item Explain how Darshan works. What is it, who created it, what does it do %exactly? What areas does it lack? (i.e. no timestamps)
% \item Explain how LDMS works (high level overview -- same as Darshan). What capabilities does it have that we will be applying to Darshan to transport the data.
% \item Explain how SOS works and why we are using it.
% \item Explain how Grafana works with SOS and why we are using it.
%\end{itemize}
%}