spark-sql-streaming-StreamingQuery.adoc

StreamingQuery Contract

StreamingQuery is the contract for a streaming query that is executed continuously and concurrently (i.e. on a separate thread).

Note	StreamingQuery is called continuous query or streaming query.

Note	StreamingQuery is a Scala trait with the only implementation being StreamExecution (and less importanly StreamingQueryWrapper for serializing a non-serializable StreamExecution).

StreamingQuery can be in two states:

• active (started)

• inactive (stopped)

If inactive, StreamingQuery may have transitioned into the state due to an StreamingQueryException (that is available under exception).

StreamingQuery tracks current state of all the sources, i.e. SourceStatus, as sourceStatuses.

There could only be a single Sink for a StreamingQuery with many Sources.

StreamingQuery can be stopped by stop or an exception.

Table 1. StreamingQuery Contract

Method	Description
awaitTermination	awaitTermination(): Unit awaitTermination(timeoutMs: Long): Boolean Used when…​FIXME
exception	exception: Option[StreamingQueryException] StreamingQueryException if the query has finished due to an exception Used when…​FIXME
explain	explain(): Unit explain(extended: Boolean): Unit Used when…​FIXME
id	id: UUID The unique identifier of the streaming query Used when…​FIXME
isActive	isActive: Boolean Used when…​FIXME
lastProgress	lastProgress: StreamingQueryProgress The last StreamingQueryProgress of the streaming query Used when…​FIXME
name	name: String The name of the query that is unique across all active queries Used when…​FIXME
processAllAvailable	processAllAvailable(): Unit Waits the streaming query until there are no data available in sources or the query has been terminated. Used when…​FIXME
recentProgress	recentProgress: Array[StreamingQueryProgress] Collection of the recent StreamingQueryProgress updates. Used when…​FIXME
runId	runId: UUID The unique identifier of the current execution of the streaming query Used when…​FIXME
sparkSession	sparkSession: SparkSession Used when…​FIXME
status	status: StreamingQueryStatus The StreamingQueryStatus of the streaming query (as StreamExecution has accumulated being a ProgressReporter while running the streaming query) Used when…​FIXME
stop	stop(): Unit Stops the execution of the streaming query Used exclusively when StreamingQueryWrapper is requested to stop the streaming query