User Guide 1.3.0
In order to use DDS in your Spark shell, you need to add it to your classpath and import the DDS core functions. Depending on the Scala version your Spark is compiled in, select the correct jar. Assuming your Spark is built with 2.10, start your Spark shell with the following parameter and import the core functionality:
spark-shell --jars spawncamping-dds-1.3.0_2.10.jarimport de.frosner.dds.core.DDS._DDS comes with a lightweight web server that serves the results and charts to your browser. It pushes JSON objects to the JavaScript front-end that will then display them using HTML, CSS and SVG. The server needs to be started once after the Spark shell has loaded. It can be used for the entire session. However, you can stop and restart it as often as you like.
Note that a visualization is served only once. So currently only one browser window is able to consume the result (the one that asks quickest). However, if you want the current visualization not to be refreshed, there is a small 
icon in the top right corner of the screen. Clicking this icon will prevent the front-end from talking to the back-end. This allows to have multiple visualizations open at once.
You can change the visualization title in the top left corner to save it for later reference or show it to your colleagues.
The server can be started by calling the start() function in the Spark shell. You can also specify the interface and port the server should listen to. To start the server listening to 192.168.0.5:8081, execute the following command:
start("192.168.0.5", 8081)The server can be stopped by calling the stop() function in the Spark shell.
To get a list of all available functions use the help() function. To get help for particular functions, you can pass the function name to help(name: String).
Besides the functions based on RDDs, DDS offers a bunch of generic plotting functions not related to Spark. They can be used to plot charts or display tables in the browser from any data source you have.
line[N](values: Seq[N])(implicit num: Numeric[N])Prints a line chart with the given values and a default label.
lines[N](labels: Seq[String], values: Seq[Seq[N]])(implicit num: Numeric[N])Prints a line chart with multiple lines. The label labels(x) corresponds to the value sequence values(x).
bar[N](values: Seq[N])(implicit num: Numeric[N])Prints a bar chart showing the given counts on an indexed axis and a default label.
bar[N](values: Seq[N], categories: Seq[String])(implicit num: Numeric[N])Prints a bar chart showing the given counts on a categorical axis with a default label. There must be as many categories as there are counts.
bars[N](labels: Seq[String], values: Seq[Seq[N]])(implicit num: Numeric[N])Prints a bar chart with multiple bars on an indexed axis. The label labels(x) corresponds to the value sequence values(x).
bars[N](labels: Seq[String], values: Seq[Seq[N]], categories: Seq[String])(implicit num: Numeric[N])Prints a bar chart with multiple bars on a categorical axis. The label labels(x) corresponds to the value sequence values(x). There must be a category for each value.
pie[K, V](keyValuePairs: Iterable[(K, V)])(implicit num: Numeric[V])Prints a pie chart with one segment for each key-value pair in the input collection. The keys are assumed to be unique (e.g. values already reduced).
histogram[N1, N2](bins: Seq[N1], frequencies: Seq[N2])(implicit num1: Numeric[N1], num2: Numeric[N2])Plots a histogram chart visualizing the given bins and frequencies. The bins are defined by their borders. To specify n bins, you need to pass n+1 borders and n frequencies.
Example:
- 5 people are between 0 and 18 years old, 10 people between 18 and 25
- bins = [0, 18, 25], frequencies = [5, 10]
show[V](sequence: Seq[V])(implicit tag: TypeTag[V])Shows the given value sequence in a table. DDS can show sequences of simple values (e.g. strings, numbers) or composite ones (collections, case classes). The values of that sequence need to have a type tag, i.e. if they are custom classes they need to be defined top level.
table(head: Seq[String], rows: Seq[Seq[Any]])Prints a table with the given row-wise data. The head(x) corresponds to each column, e.g. rows(0)(x).
If the table contains optional values (represented by Scala's Option[T] monad), they will be treated as JavaScript null by the front-end and displayed as empty cells if they are missing.
graph[ID, L](vertices: Seq[(ID, L)], edges: Iterable[(ID, ID)])Displays a graph defined by the given vertices and edges. A vertex is represented by a pair of an identifier and a label. Edges are pairs of source and target vertex identifiers.
show[V](rdd: RDD[V], sampleSize: Int = 100)(implicit tag: TypeTag[V])Prints the first lines of the given RDD in a table. The second (optional) argument determines the number of rows to show. DDS can show RDDs of simple values (e.g. strings, numbers) or composite ones (collections, case classes). The values of that RDD need to have a type tag, i.e. if they are custom classes they need to be defined top level.
show[VD, ED](graph: Graph[VD, ED], sampleSize: Int = 20)Plots a sample of the given GraphX graph. DDS will sample the graph by taking a subset of the nodes and discarding any edges that do not have both source and target node as part of the subset.
It is advised to choose small samples first because the graph layout algorithm and the SVG drawing can be quite slow for huge graphs.
bar[V](values: RDD[V])Prints a bar chart visualizing the count of all distinct values in this RDD. It is recommended to execute it only on non-numeric value RDDs. Use the histogram function for numeric RDDs instead.
pie[V](values: RDD[V])Prints a pie chart visualizing the count of all distinct values in this RDD. It is recommended to execute it only on non-numeric value RDDs. Use the histogram function for numeric RDDs instead.
histogram[N](values: RDD[N], numBuckets: Int = 100)(implicit num: Numeric[N])Prints a histogram visualizing the given numerical RDD. The second (optional) parameter specifies the number of evenly distributed buckets.
histogram[N1, N2](values: RDD[N1], buckets: Seq[N2])
(implicit num1: Numeric[N1], num2: Numeric[N2])Prints a histogram visualizing the given numerical RDD. The second parameter specifies the buckets to use. Example: To get two buckets, one from 0 to 5 and another one from 5 to 10, you can to pass buckets = List(0,5,10).
summarize[N](values: RDD[N])(implicit num: Numeric[N] = null)Shows some basic summary statistics for the given RDD. If the RDD has numeric values, count, sum, min, max, mean, stdev, and variance are calculated. If the RDD has nominal values, DDS will calculate mode and cardinality for you.
groupAndSummarize[K, N](readyToGroup: RDD[(K, N)])(implicit num: Numeric[N])Shows some basic summary statistics for each of the groups defined by the given key. It is assumed that there are key-value pairs in each input row, where the key can be used for grouping.
summarizeGroups[K, N](grouped: RDD[(K, Iterable[N])])(implicit num: Numeric[N])Shows some basic summary statistics for each of the given groups. It is assumed that there is one input row per group and is usually a result of a group-by operation on an RDD.
groupAndPie[K, N](readyToGroup: RDD[(K, N)])(reduceFunction: (N, N) => N)(implicit num: Numeric[N])Computes a pie chart visualizing the numeric values per group. It is assumed that there are key-value pairs in each input row, where the key can be used for grouping. DDS will apply the given reduce function to the values of each group before plotting the reduced value in a segment.
pieGroups[K, N](grouped: RDD[(K, Iterable[N])])(reduceFunction: (N, N) => N)(implicit num: Numeric[N])Computes a pie chart visualizing the numeric values per group. It is assumed that there is one input row per group and is usually a result of a group-by operation on an RDD. DDS will apply the given reduce function to the values of each group before plotting the reduced value in a segment.
DDS comes with a set of interactive visualizations at the front-end side. The following sections explains only the visualizations that are truly interactive, i.e. the user can modify it by clicking and pressing buttons. Simple visualizations like tables or line charts are not covered here.
Tabular data is displayed in an interactive grid. The table contents can be sorted by clicking on the column header of the attribute to sort by. Each element is assigned an ID which links the contents of the grid with the accompanying plot (e.g. parallel coordinates).
If the table has many entries, it will be paginated. You can find the controls for pagination in the menu bar above the grid.
The parallel coordinates plot is a visualization technique to explore high dimensional data. Instead of drawing the axes orthogonally like in a 2D scatter plot, they are drawn in a parallel fashion. One row / data point corresponds to a polyline through all the axes. DDS currently supports two interactions with the parallel coordinates plot: Axis reordering and filtering. DDS offers parallel coordinates for all tabular data.
To reorder the axes, drag one axis on its label to a different position. This can be helpful to detect patterns like correlations or clusters in the data.
To change the ordering of the values on the axis, double click on the axis label.
You can filter the values by selecting a range in one of the dimensions. Just click and span a filter region on an axis. DDS supports filters on multiple dimensions simultaneously. To change the filter just move it around or change its size. To delete a filter, just click on a different part of the axis that is not in range of the filter.
DDS currently uses the Apache log4j logger which is also used by Spark. So you can configure the logging in the same way you configure logging in Spark.