logging docs: mention other sinks

modules/concepts/pages/logging.adoc

@@ -1,32 +1,59 @@
 = Logging
+:description: A conceptual explanation of the logging architecture of the Stackable Data Platform, and how it is configured.
+:keywords: logging, observability, log aggregation, Kubernetes, k8s, Vector, Elasticsearch, OpenSearch
 
 // Abstract
-Logging is important for observability of the platform. Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform. Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well.
+Logging is important for observability of the platform.
+Stackable provides human-readable plaintext logs for each running container, as well as aggregated and persisted logs with identical structure across the whole platform.
+Log levels can be set for individual modules and configuration is identical across all products, but custom logging configuration files can be supplied as well.
 
 image::logging_overview.drawio.svg[]
 
 == Motivation
 
-**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts. For this, logs should share the same structure, and should be viewable in a central location. Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason.
+**Aggregate and persist** - The logging on the platform was designed to aggregate logs from all parts of the platform to make it easy to correlate events from different parts.
+For this, logs should share the same structure, and should be viewable in a central location.
+Logs should also be persisted in a central location, so if a component crashes, the logs are still there to identify the reason.
 
-**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform. The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs. This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.
+**Easy to read on the fly** - At the same time, logs should still be accessible in an easy to read format on the containers, to allow for easy on the fly inspection of each part of the platform.
+The logging configuration also supports setting different thresholds for the logs readable on the container and the aggregated logs.
+This way you can get a detailed view of the operations of a component while viewing it on the container, but aggregate logs at a coarser granularity when aggregating across the whole platform.
 
-**Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs. Logging for each product is configured in the ProductCluster resource. It is still supported to supply custom logging configuration files, these are then product specific.
+**Consistent configuration** - Finally, logging should be always configured the same way, no matter which product and which underlying technology is used to produce the logs.
+Logging for each product is configured in the ProductCluster resource.
+It is still supported to supply custom logging configuration files, these are then product specific.
 
 == Architecture
 
-Below you can see the overall architecture using ZooKeeper as an example. Stackable uses https://vector.dev/[Vector] for log aggregation and https://opensearch.org/[OpenSearch] as the sink to persist logs in.
+Below you can see the overall architecture using ZooKeeper as an example.
+Stackable uses https://vector.dev/[Vector] for log aggregation and any of the supported https://vector.dev/docs/reference/configuration/sinks/[sinks] can be used to persist the logs.
 
 image::logging_architecture.drawio.svg[]
 
-You configure your logging in the resource describing the cluster, seen in the top left in the diagram (see the <<configuration, configuration>> section below). The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration. The ConfigMap is then mounted into the containers. The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container]. All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator].
+=== Log source
 
-The aggregator is configured to use the OpenSearch sink. It sends all logs to the sink. If the sink is unavailable, it will buffer the log messages. It is also the single point where the sink is configured, so the sink is decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform.
+You configure your logging settings in the resource describing the cluster (ZookeeperCluster in this example), seen in the top left in the diagram (see the <<configuration, configuration>> section below).
+The operator reads this resource and creates the appropriate log configuration files in the ConfigMap which also holds other product configuration.
+The ConfigMap is then mounted into the containers.
+The ZooKeeper Pod has three containers: The `prepare` sidecar container, the `zookeeper` container and the `vector` https://vector.dev/docs/setup/deployment/roles/#sidecar[sidecar container].
+All logs get written into a shared mounted directory, from which the Vector agent reads them and sends them to the Vector https://vector.dev/docs/setup/deployment/roles/#aggregator[aggregator].
+
+=== Aggregator and sinks
+
+The aggregator is configured to use one or multiple https://vector.dev/docs/reference/configuration/sinks/[sinks] (for example OpenSearch, Elasticsearch), it sends all logs to all sinks.
+If a sink is unavailable, it will buffer the log messages.
+It is also the single point where the sinks are configured, so the sinks are decoupled from the individual product configurations and only needs to be configured in this single location for the whole platform.
 
 [#configuration]
-== Identical configuration in each product
+== Configuration
+
+=== Identical configuration structure in each product
 
-Logging configuration usually requires configuration what to log, where to log it and in which format. Usually you want to persist logs but also view them on the fly in a console, and possibly both of these in different formats. Stackable provides configuration for this out of the box. You only need to enable log aggregation and configure _what_ you want to log. As part of the logging configuration, individual modules can be configured with different log levels:
+Logging configuration usually requires configuration what to log, where to log it and in which format.
+Usually you want to persist logs but also view them on the fly in a console, and possibly both of these in different formats.
+Stackable provides configuration for this out of the box.
+You only need to enable log aggregation and configure _what_ you want to log.
+As part of the logging configuration, individual modules can be configured with different log levels:
 
 [source,yaml]
 ----
@@ -57,17 +84,29 @@ vectorAggregatorConfigMapName: vector-aggregator-discovery  // <1>
 <5> Console and file appenders can have different log level thresholds.
 <6> Setting log levels for individual modules is also possible.
 
-**Log levels per module** - In the `loggers` section the log levels for each module are configured. Note the `ROOT` module. This is a special module and refers to the default log level for the container. The log levels are the commonly used ones (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) as well as the NONE level to turn of logging. The default log level is `INFO`.
+**Log levels per module** - In the `loggers` section the log levels for each module are configured.
+Note the `ROOT` module.
+This is a special module and refers to the default log level for the container.
+The log levels are the commonly used ones (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) as well as the NONE level to turn of logging.
+The default log level is `INFO`.
 
-**Log levels for console and file** - The configuration supports setting another level threshold for the console and the file output from which the aggregation is done. A typical use case is that the log level for the console is set to a more detailed level to allow for detailed inspection of a container, but less information in the aggregated log store.
+**Log levels for console and file** - The configuration supports setting another level threshold for the console and the file output from which the aggregation is done.
+A typical use case is that the log level for the console is set to a more detailed level to allow for detailed inspection of a container, but less information in the aggregated log store.
 
-**Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them. Also the log levels per module are container specific.
+**Per container configuration** - A Pod actually consists of multiple containers and you might want different log levels for each of them.
+Also the log levels per module are container specific.
 
 Following the Stackable xref::roles-and-role-groups.adoc[roles and role groups] concept, this configuration fragment can be set either at role or role group level.
 
+=== Configuring the Aggregator
+
+Follow the https://vector.dev/docs/setup/installation/package-managers/helm/#aggregator[installation instructions] for the aggregator.
+Configure a https://vector.dev/docs/reference/configuration/sources/vector/[Vector source] at adress `0.0.0.0:6000` and configure sinks and additional settings according to your needs.
+
 === Configuring the aggregator location
 
-Every product also has a `vectorAggregatorConfigMapName` property in its spec that needs to be set to the name of the ConfigMap that contains the address of the aggregator. The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if you're running the aggregator behind a service named `vector-aggregator`.
+Every product also has a `vectorAggregatorConfigMapName` property in its spec that needs to be set to the name of the ConfigMap that contains the address of the aggregator.
+The field is called `ADDRESS` and the value could be `vector-aggregator:6000` if you're running the aggregator behind a service named `vector-aggregator`.
 
 == Custom overrides
 
@@ -85,6 +124,7 @@ logging:
 <1> The vector logging agent is not deployed
 <2> A custom logging configuration is loaded from a ConfigMap called `my-configmap`
 
-== Further Reading
+== Further reading
 
-To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo].
+To get some hands on experience and see logging in action, try out the xref:demos:logging.adoc[logging demo] or follow the xref:tutorials:logging-vector-aggregator.adoc[logging tutorial].
+The Vector documentation contains more information about the https://vector.dev/docs/setup/deployment/topologies/#centralized[deployment topology] and https://vector.dev/docs/reference/configuration/sinks/[sinks].

modules/tutorials/examples/logging-aggregator/main.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+echo "Installing ZooKeeper Operator"
+# tag::zk-op[]
+#stackablectl release install -i secret -i commons -i zookeeper 23.7
+stackablectl op in secret commons zookeeper
+# end::zk-op[]
+
+# tag::vector-agg[]
+helm install --wait -f vector-aggregator-values.yaml vector-aggregator vector/vector
+# end::vector-agg[]
+
+# tag::vector-discovery[]
+kubectl apply -f vector-aggregator-discovery.yaml
+# end::vector-discovery[]
+
+# tag::zk[]
+kubectl apply -f zookeeper.yaml
+# end::zk[]
+
+sleep 10
+kubectl rollout status statefulset simple-zk-server-default --timeout=5m
+kubectl wait \
+    --for=jsonpath='.status.readyReplicas'=3 \
+    --timeout=5m \
+    statefulsets.apps/simple-zk-server-default
+
+# tag::grep[]
+kubectl logs vector-aggregator-0 | grep "zookeeper.version=" | jq
+# end::grep[]
+
+if [ "${PIPESTATUS[1]}" -eq 0 ]
+then
+    echo "it worked"
+else
+    echo "it didn't work :("
+fi

modules/tutorials/examples/logging-aggregator/vector-aggregator-discovery.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: vector-aggregator-discovery
+data:
+  ADDRESS: vector-aggregator:6000

modules/tutorials/examples/logging-aggregator/vector-aggregator-values.yaml

@@ -0,0 +1,15 @@
+role: Aggregator
+customConfig:
+  sources:
+    vector:  # <1>
+      address: 0.0.0.0:6000
+      type: vector
+      version: "2"
+  sinks:
+    console:  # <2>
+      type: console
+      inputs:
+        - vector
+      encoding:
+        codec: json
+      target: stderr

modules/tutorials/examples/logging-aggregator/zookeeper.yaml

@@ -0,0 +1,32 @@
+---
+apiVersion: zookeeper.stackable.tech/v1alpha1
+kind: ZookeeperCluster
+metadata:
+  name: simple-zk
+spec:
+  image:
+    productVersion: 3.8.0
+    stackableVersion: "0.0.0-dev"
+  clusterConfig:
+    vectorAggregatorConfigMapName: vector-aggregator-discovery  # <1>
+  servers:
+    roleGroups:
+      default:
+        replicas: 3
+        config:
+          logging:  # <2>
+            enableVectorAgent: true
+            containers:
+              vector:
+                file:
+                  level: WARN
+              zookeeper:
+                console:
+                  level: INFO
+                file:
+                  level: INFO
+                loggers:
+                  ROOT:
+                    level: INFO
+                  org.apache.zookeeper.server.NettyServerCnxn:
+                    level: NONE

modules/tutorials/nav.adoc

@@ -1,4 +1,5 @@
 * xref:tutorials:index.adoc[]
 ** xref:end-to-end_data_pipeline_example.adoc[]
 ** xref:authentication_with_openldap.adoc[]
+** xref:logging-vector-aggregator.adoc[]
 ** xref:enabling_verification_of_image_signatures.adoc[]

modules/tutorials/pages/authentication_with_openldap.adoc

@@ -7,7 +7,7 @@ more about authentication in the Stackable Platform on the xref:concepts:authent
 Prerequisites:
 
 * a k8s cluster available, or https://kind.sigs.k8s.io/[kind] installed
-* xref:management::stackablectl/index.adoc installed
+* xref:management:stackablectl:index.adoc[] installed
 * basic knowledge of how to create resources in Kubernetes (i.e. `kubectl apply -f <filename>.yaml`) and inspect them
   (`kubectl get` or a tool like https://k9scli.io/[k9s])
 

modules/tutorials/pages/logging-vector-aggregator.adoc

@@ -0,0 +1,113 @@
+= Logging with a Vector log aggregator
+:description: A tutorial on how to set up a Vector log aggregator to use with the Stackable Data Platform.
+:keywords: logging, observability, log aggregation, Kubernetes, k8s, Vector, ZooKeeper
+
+This tutorial teaches you how to deploy a Vector aggregator together with a product - in this case ZooKeeper - and how to configure both of them so the logs are sent from the product to the aggregator.
+Logging on the Stackable Data Platform is always configured in the same way, so you can use this knowledge to configure logging in any product that you want to deploy.
+
+Prerequisites:
+
+* a k8s cluster available, or https://kind.sigs.k8s.io/[kind] installed
+* xref:management:stackablectl:index.adoc[] installed
+* https://helm.sh/[Helm] installed to deploy Vector
+* basic knowledge of how to create resources in Kubernetes (i.e. `kubectl apply -f <filename>.yaml`) and inspect them
+  (`kubectl get` or a tool like https://k9scli.io/[k9s])
+
+== Install the ZooKeeper operator
+
+Install the Stackable Operator for Apache ZooKeeper and its dependencies, so you can deploy a ZooKeeper instance later.
+
+[source,bash]
+include::example$logging-aggregator/main.sh[tag=zk-op]
+
+== Install the Vector aggregator
+
+Install the Vector aggregator using Helm.
+First, create a `vector-aggregator-values.yaml` file with the Helm values:
+
+[source,yaml]
+include::example$logging-aggregator/vector-aggregator-values.yaml[]
+
+<1> define a source of `type` `vector` which listens to incoming log messages at port 6000.
+<2> define a `console` sink, logging all received logs to `stderr`.
+
+Deploy Vector with these values using Helm:
+
+[source,bash]
+include::example$logging-aggregator/main.sh[tag=vector-agg]
+
+This is a minimal working configuration. The source should be defined in this way, but you can configure different sinks, depending on your needs.
+You can find an https://vector.dev/docs/reference/configuration/sinks/[overview] of all sinks in the Vector documentation, specifically the https://vector.dev/docs/reference/configuration/sinks/elasticsearch/[Elasticsearch] sink might be useful, it also works when configured with OpenSearch.
+
+To make the Vector aggregator discoverable to ZooKeeper, deploy a xref:concepts:service_discovery.adoc[discovery ConfigMap] called `vector-aggregator-discovery`.
+Create a file called `vector-aggregator-discovery.yaml`:
+
+[source,yaml]
+include::example$logging-aggregator/vector-aggregator-discovery.yaml[]
+
+and apply it:
+
+[source,bash]
+include::example$logging-aggregator/main.sh[tag=vector-discovery]
+
+== Install ZooKeeper
+
+Now that the aggregator is running, you can install a ZooKeeper cluster which is configured to send logs to the aggregator.
+
+Create a file called `zookeeper.yaml` with the following ZookeeperCluster definition:
+
+[source,yaml]
+include::example$logging-aggregator/zookeeper.yaml[]
+
+<1> This is the reference to the discovery ConfigMap created in the previous step.
+<2> This is the logging configuration, where logging is first enabled and then a few settings are made.
+
+and apply it:
+
+[source,bash]
+include::example$logging-aggregator/main.sh[tag=zk]
+
+TIP: You can learn more about how to configure logging in a product at the xref:concepts:logging.adoc[logging concept documentation].
+
+== Watch the logs
+
+During startup, ZooKeeper already prints out log messages.
+Vector was configured to print the aggregated logs to `stderr`, so if you look at the logs of the Vector pod, you will see the ZooKeeper logs:
+
+
+[source,bash]
+include::example$logging-aggregator/main.sh[tag=grep]
+
+You should see a JSON oject per ZooKeeper replica printed that look like
+
+[source,json]
+{
+  "cluster": "simple-zk",
+  "container": "zookeeper",
+  "file": "zookeeper.log4j.xml",
+  "level": "INFO",
+  "logger": "org.apache.zookeeper.server.ZooKeeperServer",
+  "message": "Server environment:zookeeper.version=3.8.0-5a02a05eddb59aee6ac762f7ea82e92a68eb9c0f, built on 2022-02-25 08:49 UTC",
+  "namespace": "default",
+  "pod": "simple-zk-server-default-0",
+  "role": "server",
+  "roleGroup": "default",
+  "source_type": "vector",
+  "timestamp": "2023-11-06T10:30:40.223Z"
+}
+
+The JSON object contains a timestamp, the log message, log level and some additional information.
+
+You can see the same log line in the log output of the ZooKeeper container:
+
+[source]
+kubectl logs --container=zookeeper simple-zk-server-default-0 | grep "zookeeper.version="
+
+[source]
+2023-11-06 10:30:40,223 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:zookeeper.version=3.8.0-5a02a05eddb59aee6ac762f7ea82e92a68eb9c0f, built on 2022-02-25 08:49 UTC
+
+Congratulations, this concludes the tutorial!
+
+== What's next?
+
+Look into different sink configurations which are more suited to production use in the https://vector.dev/docs/reference/configuration/sinks/[sinks overview documetation] or learn more about how logging works on the platform in the xref:concepts:logging.adoc[concepts documentation].