conc.tex

\clearpage

\section{Concluding remarks}\label{sec:conclusion}

Delay scheduling has long been accepted as a \textit{de facto} solution for achieving
higher data locality (and assumedly better performance) in cluster framework schedulers that
enforce fairness. We show that 
delay scheduling, using a fixed delay interval, is inefficient, and can be improved by 
adapting the delay interval on-the-fly, based on the overhead experienced by completed tasks. 
An adaptive solution allows delay scheduling to find a proper balance between fairness and
performance, based on changing network conditions, job profiles, and cluster usage.
We present Dynamic Delay Scheduling as a simple proof of concept that adaptive scheduling
in these systems can provide benefits with minimal state.

Regarding future work, there are other areas where adaptivity looks promising in the realm of
delay scheduling. For example, the frequency with which tasks complete is major contributor
to the probability that delay scheduling will actually provide locality, and could potentially
be used as another adaptive improvement. Other feedback metrics (such as task completion time) 
hold promise for improvements/heuristics as well. We plan to evaluate these adaptive approaches
with other varied workloads, such as TPC-DS. Another promising direction is the analysis of the applicability
of this adaptivity to stream processing frameworks, such as Spark Streaming~\cite{Zaharia:2013str}.