Skip to content

Commit

Permalink
Update docs (#134)
Browse files Browse the repository at this point in the history
* First set of documentation and corresponding code updates
* Update toTopicNameExtractor
* Syntax and reference updates
* Update metric documentation and several related subjects
* Remove stores declaration from operations, which was a duplicate of functions declaring state store names
* Only parse store names for functions that can use state stores
* Update function documentation
  • Loading branch information
jeroenvandisseldorp authored Jun 28, 2024
1 parent 647908d commit d3566a2
Show file tree
Hide file tree
Showing 65 changed files with 1,458 additions and 857 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ To run the KSML demo, we provide a Docker compose file which will start Kafka, c
container
with a demo producer. You can then start the runner you generated in the previous step, passing in a KSML configuration
of your choice.
See [Getting started](docs/getting-started.md) or [Runners](docs/runners.md) for details.
See [Getting started](docs/quick-start) or [Runners](docs/runners.md) for details.

To run the demo, Docker 19.x is required.

Expand Down
2 changes: 1 addition & 1 deletion docs/_data/menu.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ items:
url: runners.html

- title: Language Specification
url: specification.html
url: ksml-language-spec.html
361 changes: 323 additions & 38 deletions docs/functions.md

Large diffs are not rendered by default.

32 changes: 14 additions & 18 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,25 @@

Welcome to the KSML documentation, Use the menu on the left to navigate through the various sections

## Introduction
KSML allows anyone to specify a powerful Kafka Streams application in just a few lines of YAML and Python snippets.
## Quick Start

## Prerequisites to run KSML
- Docker 19.03.0 or newer
- Kafka cluster 0.11 or newer
- Access to Docker Hub to pull images (internet connectivity)
If you want to get going quickly, go to the KSML Quickstart.

## Prerequisites to build KSML
- GraalVM JDK 21.0.2 with Java 21
- Maven 3 - To build the project
## Introduction

KSML allows anyone to specify a powerful Kafka Streams application in just a few lines of YAML and Python snippets.

## Contents

1. [Introduction](introduction.md)
1. [Stream Types](streams.md)
1. [Functions](functions.md)
1. [Pipelines](pipelines.md)
1. [Operations](operations.md)
1. [Data Types](types.md)
1. [Runners](runners.md)
1. [Language specification](ksml-language-spec)

[Getting Started](getting-started.md)
2. [Stream Types](streams.md)
3. [Functions](functions.md)
4. [Pipelines](pipelines.md)
5. [Operations](operations.md)
6. [Data Types](types.md)
7. [Runners](runners.md)
8. [Language specification](ksml-language-spec.md)

[Getting Started](quick-start)

[Release Notes](release-notes.md)
127 changes: 88 additions & 39 deletions docs/introduction.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/ksml-language-spec.json

Large diffs are not rendered by default.

201 changes: 81 additions & 120 deletions docs/ksml-language-spec.md

Large diffs are not rendered by default.

56 changes: 36 additions & 20 deletions docs/notations.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,62 @@
# Notations

### Table of Contents

1. [Introduction](#introduction)
1. [Avro](#avro)
1. [CSV](#csv)
1. [JSON](#json)
1. [SOAP](#soap)
1. [XML](#xml)
2. [Avro](#avro)
3. [CSV](#csv)
4. [JSON](#json)
5. [SOAP](#soap)
6. [XML](#xml)

## Introduction

KSML is able to express its internal data types in a number of external representations. Internally these are called _notations_.
The different notations are described below.
KSML is able to express its internal data types in a number of external representations. Internally these are called
_notations_. The different notations are described below.

## AVRO

Avro types are supported through the "avro" prefix in types. The notation is ```avro:schema```, where schema is the schema fqdn, or just the schema name itself.
Avro types are supported through the "avro" prefix in types. The notation is ```avro:schema```, where schema is the
schema fqdn, or just the schema name itself.

On Kafka topics, Avro types are serialized in binary format. Internally they are represented as structs.

Examples
Examples:

```
avro:SensorData
avro:io.axual.ksml.example.SensorData
```

Note: when referencing an AVRO schema, you have to ensure that the respective schema file can be found in the KSML working directory and has the .avsc file extension.
Note: when referencing an AVRO schema, you have to ensure that the respective schema file can be found in the KSML
working directory and has the .avsc file extension.

## CSV

Comma-separated values are supported through the "csv" prefix in types. The notation is ```csv:schema```, where schema is the schema fqdn, or just the schema name itself.
Comma-separated values are supported through the "csv" prefix in types. The notation is ```csv:schema```, where schema
is the schema fqdn, or just the schema name itself.

On Kafka topics, CSV types are serialized as `string`. Internally they are represented as structs.

Examples
Examples:

```
csv:SensorData
csv:io.axual.ksml.example.SensorData
```

Note: when referencing an CSV schema, you have to ensure that the respective schema file can be found in the KSML working directory and has the .csv file extension.
Note: when referencing an CSV schema, you have to ensure that the respective schema file can be found in the KSML
working directory and has the .csv file extension.

## JSON

JSON types are supported through the "json" prefix in types. The notation is ```json:schema```, where `schema` is the schema fqdn, or just the schema name itself.
JSON types are supported through the "json" prefix in types. The notation is ```json:schema```, where `schema` is the
schema fqdn, or just the schema name itself.

On Kafka topics, JSON types are serialized as `string`. Internally they are represented as structs or lists.

Examples
Examples:

```
json:SensorData
json:io.axual.ksml.example.SensorData
Expand All @@ -58,20 +67,27 @@ If you want to use JSON without a schema, you can leave out the colon and schema
```
json
```
Note: when referencing an JSON schema, you have to ensure that the respective schema file can be found in the KSML working directory and has the .json file extension.

Note: when referencing an JSON schema, you have to ensure that the respective schema file can be found in the KSML
working directory and has the .json file extension.

## SOAP

SOAP is supported through built-in serializers and deserializers. The representation on Kafka will always be ```string```. Internally SOAP objects are structs with their own schema. Field names are derived from the SOAP standards.
SOAP is supported through built-in serializers and deserializers. The representation on Kafka will always
be ```string```. Internally SOAP objects are structs with their own schema. Field names are derived from the SOAP
standards.

## XML

XML is supported through built-in serializers and deserializers. The representation on Kafka will always be ```string```. Internally XML objects are structs.
XML is supported through built-in serializers and deserializers. The representation on Kafka will always
be ```string```. Internally XML objects are structs.

Examples:

Examples
```
xml:SensorData
xml:io.axual.ksml.example.SensorData
```

Note: when referencing an XML schema, you have to ensure that the respective schema file can be found in the KSML working directory and has the .xsd file extension.
Note: when referencing an XML schema, you have to ensure that the respective schema file can be found in the KSML
working directory and has the .xsd file extension.
Loading

0 comments on commit d3566a2

Please sign in to comment.