SUMMARY.adoc

Summary

1. Introduction

2. Spark Structured Streaming — Streaming Datasets

Developing Streaming Applications

1. DataStreamReader — Loading Data from Streaming Data Source

2. DataStreamWriter — Writing Datasets To Streaming Data Sinks

   1. ForeachWriter

   2. OutputMode

   3. Trigger — How Frequently to Check Sources For New Data

3. StreamingQuery

4. Streaming Operators / Streaming Dataset API

   1. dropDuplicates Operator — Streaming Deduplication

   2. explain Operator — Explaining Query Plan

   3. groupBy Operator — Untyped Streaming Aggregation (with Implicit State Logic)

   4. groupByKey Operator — Streaming Aggregation (with Explicit State Logic)

   5. withWatermark Operator — Event Time Watermark

5. window Function — Stream Time Windows

6. KeyValueGroupedDataset — Streaming Aggregation

   1. mapGroupsWithState Operator — Stateful Streaming Aggregation (with Explicit State Logic)

   2. flatMapGroupsWithState Operator — Arbitrary Stateful Streaming Aggregation (with Explicit State Logic)

7. GroupState — State Per Group in Stateful Streaming Aggregation

   1. GroupStateImpl

   2. GroupStateTimeout

8. StreamingQueryManager — Streaming Query Management

9. Configuration Properties

Kafka Data Source

1. KafkaSourceProvider — Data Source Provider for Apache Kafka

2. KafkaSource

3. KafkaRelation

4. KafkaSourceRDD

5. CachedKafkaConsumer

6. KafkaOffsetReader

7. ConsumerStrategy Contract for KafkaConsumer Providers

8. KafkaSourceOffset

9. KafkaSink

Text File Data Source

1. FileStreamSource

2. FileStreamSink — Streaming Sink for Parquet Format

Rate Source

1. RateSourceProvider

2. RateStreamSource

Text Socket Source

1. TextSocketSourceProvider

2. TextSocketSource

Memory Data Sink

1. MemorySink

2. MemoryStream

Console Data Sink

1. ConsoleSinkProvider

2. ConsoleSink for Showing DataFrames to Console

Query Planning and Execution

1. StreamExecution — Base of Streaming Query Executions

   1. MicroBatchExecution

   2. ContinuousExecution

   3. StreamingQueryWrapper

2. ProgressReporter Contract

3. QueryExecutionThread

4. TriggerExecutor

5. IncrementalExecution — QueryExecution of Streaming Datasets

6. StatefulOperatorStateInfo

7. StreamingQueryListenerBus — Notification Bus for Streaming Events

Logical Operators

1. EventTimeWatermark Unary Logical Operator

2. FlatMapGroupsWithState Unary Logical Operator

3. Deduplicate Unary Logical Operator

4. MemoryPlan Logical Query Plan

5. StreamingRelation Leaf Logical Operator for Streaming Source

6. StreamingExecutionRelation Leaf Logical Operator for Streaming Source At Execution

Physical Operators

1. EventTimeWatermarkExec Unary Physical Operator for Accumulating Event Time Watermark

   1. EventTimeStatsAccum Accumulator

2. FlatMapGroupsWithStateExec Unary Physical Operator

3. StateStoreRestoreExec Unary Physical Operator — Restoring State of Streaming Aggregates

4. StateStoreSaveExec Unary Physical Operator — Saving State of Streaming Aggregates

   1. Demo: StateStoreSaveExec with Complete Output Mode

   2. Demo: StateStoreSaveExec with Update Output Mode

5. StreamingDeduplicateExec Unary Physical Operator for Streaming Deduplication

6. StreamingRelationExec Leaf Physical Operator

7. StreamingSymmetricHashJoinExec

8. WatermarkSupport Contract for Streaming Watermark in Unary Physical Operators

Execution Planning Strategies

1. FlatMapGroupsWithStateStrategy Execution Planning Strategy for FlatMapGroupsWithState Logical Operator

2. StatefulAggregationStrategy Execution Planning Strategy for EventTimeWatermark and Aggregate Logical Operators

3. StreamingDeduplicationStrategy Execution Planning Strategy for Deduplicate Logical Operator

4. StreamingRelationStrategy Execution Planning Strategy for StreamingRelation and StreamingExecutionRelation Logical Operators

Offsets and Checkpointing

1. Offset

2. MetadataLog — Contract for Metadata Storage

3. HDFSMetadataLog — MetadataLog with Hadoop HDFS for Reliable Storage

   1. CommitLog

   2. CompactibleFileStreamLog

   3. OffsetSeqLog — HDFSMetadataLog with OffsetSeq Metadata

   4. OffsetSeq

4. OffsetSeqMetadata

Managing State in Stateful Streaming Aggregations

1. StateStore — Streaming Aggregation State Management

   1. StateStoreOps — Implicits Methods for Creating StateStoreRDD

   2. StateStoreProvider

   3. StateStoreUpdater

   4. StateStoreWriter — Recording Metrics For Writing to StateStore

2. HDFSBackedStateStore

   1. HDFSBackedStateStoreProvider

3. StateStoreRDD — RDD for Updating State (in StateStores Across Spark Cluster)

4. StateStoreCoordinator — Tracking Locations of StateStores for StateStoreRDD

   1. StateStoreCoordinatorRef Interface for Communication with StateStoreCoordinator

Monitoring

1. StreamingQueryListener — Intercepting Streaming Events

   1. StreamingQueryProgress

2. Web UI

3. Logging

Extending Structured Streaming

1. DataSource — Pluggable Data Source

2. StreamSourceProvider — Streaming Data Source Provider

3. Streaming Source

4. StreamSinkProvider

5. Streaming Sink — Adding Batches of Data to Storage

   1. ForeachSink

Structured Streaming V2

1. StreamWriteSupport

Demos

1. groupBy Streaming Aggregation with Append Output Mode

2. Developing Custom Streaming Sink (and Monitoring SQL Queries in web UI)

3. current_timestamp Function For Processing Time in Streaming Queries

4. Using StreamingQueryManager for Query Termination Management

Varia

1. UnsupportedOperationChecker

2. StreamProgress Custom Scala Map