spark-sql-streaming-StreamingRelationV2.adoc

StreamingRelationV2 Leaf Logical Operator

StreamingRelationV2 is a MultiInstanceRelation leaf logical operator that represents MicroBatchReadSupport or ContinuousReadSupport streaming data sources in a logical plan of a streaming query.

Tip	Read up on Leaf logical operators in The Internals of Spark SQL book.

StreamingRelationV2 is created when:

• DataStreamReader is requested to "load" data as a streaming DataFrame for MicroBatchReadSupport and ContinuousReadSupport streaming data sources

• ContinuousMemoryStream is created

isStreaming flag is always enabled (i.e. true).

scala> :type sq
org.apache.spark.sql.DataFrame

import org.apache.spark.sql.execution.streaming.StreamingRelationV2
val relation = sq.queryExecution.logical.asInstanceOf[StreamingRelationV2]
assert(relation.isStreaming)

StreamingRelationV2 is resolved (replaced) to the following leaf logical operators:

• ContinuousExecutionRelation when ContinuousExecution stream execution engine is requested for the analyzed logical plan

• StreamingExecutionRelation when MicroBatchExecution stream execution engine is requested for the analyzed logical plan

Creating StreamingRelationV2 Instance

StreamingRelationV2 takes the following to be created:

• DataSourceV2

• Name of the data source

• Options (Map[String, String])

• Output attributes (Seq[Attribute])

• Optional StreamingRelation

• SparkSession