JSON Format for Graphs

JSON Format for Graphs

A Topology contains a representation of its graph built using GraphBuilder. A graph is a collection of operators and the connections between them. GraphBuilder produces a JSON representation of the graph which is then transformed into SPL (from the JSON) using SPLGenerator.

JSON is used as an intermediary to allow other languages to produce a JSON representation of a graph and then utilise the Java code to generate SPL and submit to an instance, etc.

Graph (top-level) object

Attribute	Type	Value
name	string	Name of the topology/graph
namespace	string	Namespace of the topology/graph
public	boolean	Is the graph public (true for top-level graphs)
config	object	Graph configuration object
operators	array	Array of Operator objects (nodes) in the graph
parameters	object	Object with key,value pairs with key being the parameter name, value is a Parameter

Graph configuration object

Attribute	Type	Value
topology.vmArgs	array of strings	Additional Java virtual machine arguments for Java operators

Operator object

Attribute	Type	Value
name	string	name of the operator, needs to be unique within the graph
kind	string	SPL kind of the operator
sourcelocation	object	JSON object with information about the original source locations (optional)
parallelOperator	boolean	True if the operator is parallelized (optional)
width	any	width of parallel region (required if parallelOperator is true). May be a number or a SubmissionParameter.
partitioned	boolean	true if parallel partitioned
parallelInputPortName	string	partitioning port for parallel (required if parallelOperator is true)
outputs	array	Output ports objects (optional)
inputs	array	Input port objects (optional)
runtime	string	Runtime for operator (if known) (optional): spl.java SPL Java primitive
parameters	object	Object with key,value pairs with key being the parameter name, value is a Parameter value object
config	object	Operator configuration object
lowlatencyTag	string	Operators with the same id are part of the same low latency region and must be fused together. (optional) (pending rename to lowLatencyRegion and move to Operator Placement object)
autonomous	boolean	True if the operator starts a autonomous region (optional).

Output Port object

Attribute	Type	Value
type	string	SPL type of the output (e.g.tuple<int32 a rstring b>)
type.native	string	Optional Native language (Java/Scala) type when known.
name	string	name of the output port
connections (??)	string array	Names of input ports connected to input port

Input Port object

Attribute	Type	Value
type (??)	string	SPL type of the input (e.g.tuple<int32 a rstring b>)
name	string	name of the input port
connections	string array	Names of output ports connected to input port
window	object	Window input port object for input port (optional)
queue	object	Queue input port object (optional) - not present means no input port queue

Window Input Port object

Attribute	Type	Value
type	string	StreamWindow.Type name including NOT_WINDOWED
evictPolicy	string	Evict StreamWindow.Policy name (optional if NOT_WINDOWED)
evictConfig	number	Time in milliseconds or count of tuples
triggerPolicy	string	Trigger StreamWindow.Policy name (required if SLIDING)
triggerConfig	number	Time in milliseconds or count of tuples

Queue Input Port object

If an input port has a queue then it is also threaded. Currently only default queue size & number of threads (1) is supported, but future may add multi-threading and configurable queue size.

Attribute	Type	Value
functional	boolean	True if this is for a functional operator. if true then there will be a parameter queueSize for the operator (assumes a single input port at the moment).

Parameter value object

Attribute	Type	Value
value	any	Value of the parameter
type	string	Optional type for the value. Present when value requires a non-default interpretation. For a string value, type can be one of: enum, spltype, attribute, USTRING (utf16). For a Byte,Short,Integer,Long value, type can be UINT8,UINT16,UINT32,UINT64 respectively. For a object, type can be: submissionParameter for SubmissionParameter value.

SubmissionParameter parameter value object

Attribute	Type	Value
name	string	The submission parameter name.
metaType	string	The type of the submission parameter. One of: RSTRING (utf8), USTRING (utf16), BOOLEAN, FLOAT32, FLOAT64, INT8, INT16, INT32, INT64, UINT8, UINT16, UINT32, UINT64.
defaultValue	any	Optional default value for the parameter whose type is appropriate for the metaType.

Source code location object

Attribute	Type	Value
file	string	Optional source file name
class	string	Optional class name
method	string	Optional method name (Java/scala)
line	string	Optional line number in file
topology.method	string	method name for the application api

Operator Configuration object

Attribute	Type	Value
streamViewability	boolean	Can the stream be viewed by external applications/admin.
placement	object	Operator placement object

Operator Placement object

Attribute	Type	Value
explicitColocate	string	Operators with the same value must be fused together.
isolateRegion	string	Operators with the same value are part of the same isolation region. Operators from one region must not be fused with operators in another region.
resourceTags	string array	The operator must be placed in a container that will be associated with a resource that has all of the specified tags.