From 9b68169e52412124933d2cb7da40f9b5af0456b6 Mon Sep 17 00:00:00 2001 From: JENNIFER WORTHINGTON <37708067+jenworthington@users.noreply.github.com> Date: Tue, 23 Apr 2024 22:18:01 -0700 Subject: [PATCH 1/9] Update getting-started-runtime-configuration.md A lot of changes, with likely more to go. Take a look and let me know if you have comments, suggestions, etc., and I will do another edit. --- .../getting-started-runtime-configuration.md | 119 ++++++++---------- 1 file changed, 49 insertions(+), 70 deletions(-) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index 5ae63ff3b3..5bbbcfdd98 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -3,11 +3,12 @@ ## Prerequisites -* Linux host with Docker (CE 19.x or greater) or Podman enabled, depending on runtime choice (below). -* A network load balancer (NLB) configured for round-robin. Note: Special consideration may be required when more advanced products are used. -The optimal configuration of the load balancer will round-robin each http POST request (not each connection). -* The host linux OS receive buffer size should be tuned to match the sc4s default to avoid dropping events (packets) at the network level. -The default receive buffer for sc4s is set to 16 MB for UDP traffic, which should be OK for most environments. To set the host OS kernel to +* Linux host with either Docker CE version 19.x or greater or Podman enabled, depending on your runtime choice. +* A network load balancer (NLB) configured to round-robin each HTTP POST request. + +## Tune your reciever buffer +The host linux OS receiver buffer size must be tuned to match the SC4S default. This helps to avoid event dropping at the network level. +The default receiver buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to match this, edit `/etc/sysctl.conf` using the following whole-byte values corresponding to 16 MB: ```bash @@ -18,38 +19,31 @@ and apply to the kernel: ```bash sysctl -p ``` -* Ensure the kernel is not dropping packets by periodically monitoring the buffer with the command +* Verify that the kernel does not drop packets by periodically monitoring the buffer using the command `netstat -su | grep "receive errors"`. -* NOTE: Failure to account for high-volume traffic (especially UDP) by tuning the kernel will result in message loss, which can be _very_ +* Failure to tune the kernel for high-volume traffic results in message loss, which can be unpredictable and difficult to detect. See this helpful discussion in the syslog-ng [Professional Edition](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-premium-edition/7.0.10/collecting-log-messages-from-udp-sources) documentation regarding tuning syslog-ng in particular (via the [SC4S_SOURCE_*_SO_RCVBUFF](../configuration.md#syslog-source-configuration) -environment variable in sc4s) as well as overall host kernel tuning. The default values for receive kernel buffers in most distros is 2 MB, -which has proven inadequate for many. - -## IPv4 Forwarding +environment variable in SC4S) as well as overall host kernel tuning. The default values for receiver kernel buffers in most distrobutions is 2 MB, +which may not be adequate for your configuration. -In many distributions (e.g. CentOS provisioned in AWS), IPV4 forwarding is _not_ enabled by default. -This needs to be enabled for container networking to function properly. The following is an example -to check and set this up; as usual this needs to be vetted with your enterprise security policy: +## Configure IPv4 forwarding -To check: +In many distributions, for example CentOS provisioned in AWS, IPv4 forwarding is not enabled by default. +IPv4 forwarding must be enabled for container networking. +* To check that IPv4 forwarding is enabled: ```sudo sysctl net.ipv4.ip_forward``` -To set: +* To enable IPv4 forwarding: ```sudo sysctl net.ipv4.ip_forward=1``` -To ensure the change survives a reboot: - -* sysctl settings are defined through files in ```/usr/lib/sysctl.d/```, ```/run/sysctl.d/```, and ```/etc/sysctl.d/```. -* To override only specific settings, you can either add a file with a lexically later name in ```/etc/sysctl.d/``` and put following setting there: +* To ensure your changes persist upon reboot: + * Define sysctl settings through files in ```/usr/lib/sysctl.d/```, ```/run/sysctl.d/```, and ```/etc/sysctl.d/```. + * To override only specific settings, either add a file with a lexically later name in ```/etc/sysctl.d/``` and put following setting there or find this specific setting in one of the existing configuration files and set the value to ```1```. ``` net.ipv4.ip_forward=1 ``` -* or find this specific setting in one of existing configuration files (mentioned above) and set value to ```1```. -``` -net.ipv4.ip_forward=1 -``` ## Select a Container Runtime and SC4S Configuration | Container Runtime and Orchestration | Operating Systems | @@ -66,64 +60,49 @@ net.ipv4.ip_forward=1 | [Ansible+Docker](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 18.04LTS(and higher) | -### Docker and Podman basic configurations -* To run properly sc4s you need to create directories:`/opt/sc4s/local` `/opt/sc4s/archive` `/opt/sc4s/tls` -* `/opt/sc4s/local` will be used as a mount point for local overrides and configurations. -The empty `local` directory created above will populate with defaults and examples at the first invocation -of SC4S for local configurations and context overrides. _Do not_ change the directory structure of -the files that are laid down; change (or add) only individual files if desired. SC4S depends on the directory layout -to read the local configurations properly. See the notes below for which files will be preserved on restarts. -In the `local/config/` directory there are four subdirectories that allow you to provide support for device types -that are not provided out of the box in SC4S. To get you started, there is an example log path template (`lp-example.conf.tmpl`) -and a filter (`example.conf`) in the `log_paths` and `filters` subdirectories, respectively. These should _not_ be used directly, -but copied as templates for your own log path development. They _will_ get overwritten at each SC4S start. -In the `local/context` directory, if you change the "non-example" version of a file (e.g. `splunk_metadata.csv`) the changes -will be preserved on a restart. +### Create directories for your Docker or Podman configuration -* `/opt/sc4s/archive` will be used as a mount point for local storage of syslog events -(if the optional mount is uncommented above). The events will be written in the syslog-ng EWMM format. See the "configuration" -document for details on the directory structure the archive uses. +Create the following three directories: +* `/opt/sc4s/local`: This directory is used as a mount point for local overrides and configurations. This empty `local` directory populates with defaults and examples at the first invocation +of SC4S for local configurations and context overrides. Do not change the directory structure of +these files, as SC4S depends on the directory layout +to read the local configurations properly. If necessary, you can change or add individual files. + * In the `local/config/` directory four subdirectories let you provide support for device types +that are not provided out of the box in SC4S. To get started, see the example log path template `lp-example.conf.tmpl` +and a filter `example.conf` in the `log_paths` and `filters` subdirectories. Copy these as templates for your own log path development. + * In the `local/context` directory, change the "non-example" version of a file (e.g. `splunk_metadata.csv`) to preserve the changes +upon restart. +* `/opt/sc4s/archive` is a mount point for local storage of syslog events +if the optional mount is uncommented. The events are written in the syslog-ng EWMM format. See the [Configuration](https://splunk.github.io/splunk-connect-for-syslog/main/configuration/) +topic for information about the directory structure that the archive uses. -* `/opt/sc4s/tls` will be used as a mount point for custom TLS certificates -(if the optional mount is uncommented above). +* `/opt/sc4s/tls` is a mount point for custom TLS certificates if the optional mount is uncommented. -* IMPORTANT: When creating the directories above, ensure the directories created match the volume mounts specified in the -sc4s.service [unit file](podman-systemd-general.md#unit-file). Failure to do this will cause SC4S to abort at startup. + * When you create these directories, make sure that they match the volume mounts specified in the +sc4s.service [unit file](podman-systemd-general.md#unit-file). Failure to do this will cause SC4S to abort at startup. -#### Dedicated (Unique) Listening Ports +#### Configure unique dedicated listening ports -For certain source technologies, categorization by message content is impossible due to the lack of a unique "fingerprint" in -the data. In other cases, a unique listening port is required for certain devices due to network requirements in the enterprise. -For collection of such sources, we provide a means of dedicating a unique listening port to a specific source. +You can dedicate a unique listening port to a specific source. You may need to do this because, for example, certain source technologies cannot categorize by message content, or because your configuration requires a unique listening port for certain devices. -Follow this step to configure unique ports for one or more sources: - -* Modify the `/opt/sc4s/env_file` file to include the port-specific environment variable(s). Refer to the ["Sources"](../sources/index.md) -documentation to identify the specific environment variables that are mapped to each data source vendor/technology. +To configure unique dedicated ports for one or more sources, modify the `/opt/sc4s/env_file` file to include the port-specific environment variable(s). See ["Sources"](../sources/index.md) to identify the specific environment variables that are mapped to each data source vendor or technology. #### Modify index destinations for Splunk -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most customers. - -* If changes need to be made to index destinations, navigate to the `/opt/sc4s/local/context` directory to start. -* Edit `splunk_metadata.csv` to review or change the index configuration as required for the data sources utilized in your -environment. The key (1st column) in this file uses the syntax `vendor_product`. Simply replace the index value (the 3rd column) in the -desired row with the index appropriate for your Splunk installation. The "Sources" document details the specific `vendor_product` keys (rows) -in this table that pertain to the individual data source filters that are included with SC4S. -* Other Splunk metadata (e.g. source and sourcetype) can be overridden via this file as well. This is an advanced topic, and further -information is covered in the "Log Path overrides" section of the Configuration document. +Log paths are preconfigured to utilize a convention of index destinations that are suitable for most configuratons. To make changes to index destinations: +1. Navigate to `/opt/sc4s/local/context`. +2. Edit `splunk_metadata.csv` to review or change the index configuration for the data sources in your +environment. The key column uses the syntax `vendor_product`. Replace the index value in the third column of the +desired row with the index for your Splunk installation. See [Sources](https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/AVI/document) for specific `vendor_product` keys +that pertain to the individual data source filters that are included with SC4S. +3. Splunk metadata such as source and sourcetype can also be overridden using this file. This is an advanced topic and is covered in the [Log Path overrides](https://splunk.github.io/splunk-connect-for-syslog/main/configuration/) section of the Configuration topic. -#### Configure source filtering by source IP or host name +#### Configure source filtering by source IP address or host name -Legacy sources and non-standard-compliant sources require configuration by source IP or hostname as included in the event. The following steps -apply to support such sources. To identify sources that require this step, refer to the "sources" section of this documentation. See documentation -for your vendor/product to determine if specific configuration is required +You must configure legacy and non-standard-compliant sources by source IP or hostname as included in the event. To identify sources that require this step, see ["Sources"](../sources/index.md). #### Configure compliance index/metadata overrides -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. -The two files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` can be used for this purpose. These operate similarly to -the files above, where the `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that can be overridden based on the filter name. This is an advanced topic, and further information is -covered in the "Override index or metadata based on host, ip, or subnet" section of the Configuration document. +In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. You use the files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` to do this. The `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file +lists one or more metadata items that you can override based on the filter name. This is an advanced topic, see [Override index or metadata based on host, IP, or subnet]{https://splunk.github.io/splunk-connect-for-syslog/main/configuration/#override-index-or-metadata-based-on-host-ip-or-subnet-compliance-overrides). From fdd02ff48c36cc19c648003bbfc4eab610cf4c9c Mon Sep 17 00:00:00 2001 From: mstopa-splunk Date: Mon, 29 Apr 2024 14:57:10 +0000 Subject: [PATCH 2/9] Refactor documentation --- docs/{gettingstarted => }/create-parser.md | 0 .../getting-started-runtime-configuration.md | 90 ++++++------------- docs/gettingstarted/index.md | 2 +- docs/gettingstarted/quickstart_guide.md | 1 + mkdocs.yml | 4 +- 5 files changed, 33 insertions(+), 64 deletions(-) rename docs/{gettingstarted => }/create-parser.md (100%) diff --git a/docs/gettingstarted/create-parser.md b/docs/create-parser.md similarity index 100% rename from docs/gettingstarted/create-parser.md rename to docs/create-parser.md diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index 5bbbcfdd98..a8a8fc67fa 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -1,34 +1,31 @@ # Implement a Container Runtime and SC4S -## Prerequisites - -* Linux host with either Docker CE version 19.x or greater or Podman enabled, depending on your runtime choice. -* A network load balancer (NLB) configured to round-robin each HTTP POST request. - -## Tune your reciever buffer +## OS-level configuration +### Tune your receiver buffer The host linux OS receiver buffer size must be tuned to match the SC4S default. This helps to avoid event dropping at the network level. The default receiver buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to -match this, edit `/etc/sysctl.conf` using the following whole-byte values corresponding to 16 MB: +match this: +1. Edit `/etc/sysctl.conf` using the following whole-byte values corresponding to 16 MB: ```bash net.core.rmem_default = 17039360 net.core.rmem_max = 17039360 ``` -and apply to the kernel: + +2. Apply to the kernel: ```bash sysctl -p ``` -* Verify that the kernel does not drop packets by periodically monitoring the buffer using the command + +3. Verify that the kernel does not drop packets by periodically monitoring the buffer using the command `netstat -su | grep "receive errors"`. + * Failure to tune the kernel for high-volume traffic results in message loss, which can be -unpredictable and difficult to detect. See this helpful discussion in the syslog-ng -[Professional Edition](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-premium-edition/7.0.10/collecting-log-messages-from-udp-sources) -documentation regarding tuning syslog-ng in particular (via the [SC4S_SOURCE_*_SO_RCVBUFF](../configuration.md#syslog-source-configuration) -environment variable in SC4S) as well as overall host kernel tuning. The default values for receiver kernel buffers in most distrobutions is 2 MB, +unpredictable and difficult to detect. The default values for receiver kernel buffers in most distributions is 2 MB, which may not be adequate for your configuration. -## Configure IPv4 forwarding +### Configure IPv4 forwarding In many distributions, for example CentOS provisioned in AWS, IPv4 forwarding is not enabled by default. IPv4 forwarding must be enabled for container networking. @@ -40,33 +37,16 @@ IPv4 forwarding must be enabled for container networking. * To ensure your changes persist upon reboot: * Define sysctl settings through files in ```/usr/lib/sysctl.d/```, ```/run/sysctl.d/```, and ```/etc/sysctl.d/```. * To override only specific settings, either add a file with a lexically later name in ```/etc/sysctl.d/``` and put following setting there or find this specific setting in one of the existing configuration files and set the value to ```1```. + ``` net.ipv4.ip_forward=1 ``` -## Select a Container Runtime and SC4S Configuration - -| Container Runtime and Orchestration | Operating Systems | -|-------------------------------------------------------------------|-------------------------------------------------------------------------------------| -| [MicroK8s](k8s-microk8s.md) | Ubuntu with Microk8s | -| [Podman 1.7 & 1.9 + systemd](podman-systemd-general.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | -| [Docker CE 19 (and greater) + systemd](docker-systemd-general.md) | RHEL or CentOS >7.7 (best option), Debian or Ubuntu 18.04LTS | -| [Docker Desktop + Compose](docker-compose-MacOS.md) | MacOS | -| [Docker Desktop + Compose](docker-compose.md) | RHEL or CentOS 8.1 & 8.2 (best option) | -| [Bring your own Environment](byoe-rhel8.md) | RHEL or CentOS 8.1 & 8.2 (best option) | -| [Offline Container Installation](docker-podman-offline.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | -| [Ansible+Docker Swarm](ansible-docker-swarm.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | -| [Ansible+Podman](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 20.10LTS(and higher) | -| [Ansible+Docker](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 18.04LTS(and higher) | - - -### Create directories for your Docker or Podman configuration +## Create local directory structure Create the following three directories: -* `/opt/sc4s/local`: This directory is used as a mount point for local overrides and configurations. This empty `local` directory populates with defaults and examples at the first invocation -of SC4S for local configurations and context overrides. Do not change the directory structure of -these files, as SC4S depends on the directory layout -to read the local configurations properly. If necessary, you can change or add individual files. + +* `/opt/sc4s/local`: This directory is used as a mount point for local overrides and configurations. This empty `local` directory populates with defaults and examples at the first invocation of SC4S for local configurations and context overrides. Do not change the directory structure of these files, as SC4S depends on the directory layout to read the local configurations properly. If necessary, you can change or add individual files. * In the `local/config/` directory four subdirectories let you provide support for device types that are not provided out of the box in SC4S. To get started, see the example log path template `lp-example.conf.tmpl` and a filter `example.conf` in the `log_paths` and `filters` subdirectories. Copy these as templates for your own log path development. @@ -75,34 +55,22 @@ upon restart. * `/opt/sc4s/archive` is a mount point for local storage of syslog events if the optional mount is uncommented. The events are written in the syslog-ng EWMM format. See the [Configuration](https://splunk.github.io/splunk-connect-for-syslog/main/configuration/) topic for information about the directory structure that the archive uses. - * `/opt/sc4s/tls` is a mount point for custom TLS certificates if the optional mount is uncommented. - * When you create these directories, make sure that they match the volume mounts specified in the +When you create these directories, make sure that they match the volume mounts specified in the sc4s.service [unit file](podman-systemd-general.md#unit-file). Failure to do this will cause SC4S to abort at startup. +## Select a Container Runtime and SC4S Configuration -#### Configure unique dedicated listening ports - -You can dedicate a unique listening port to a specific source. You may need to do this because, for example, certain source technologies cannot categorize by message content, or because your configuration requires a unique listening port for certain devices. - -To configure unique dedicated ports for one or more sources, modify the `/opt/sc4s/env_file` file to include the port-specific environment variable(s). See ["Sources"](../sources/index.md) to identify the specific environment variables that are mapped to each data source vendor or technology. - -#### Modify index destinations for Splunk - -Log paths are preconfigured to utilize a convention of index destinations that are suitable for most configuratons. To make changes to index destinations: -1. Navigate to `/opt/sc4s/local/context`. -2. Edit `splunk_metadata.csv` to review or change the index configuration for the data sources in your -environment. The key column uses the syntax `vendor_product`. Replace the index value in the third column of the -desired row with the index for your Splunk installation. See [Sources](https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/AVI/document) for specific `vendor_product` keys -that pertain to the individual data source filters that are included with SC4S. -3. Splunk metadata such as source and sourcetype can also be overridden using this file. This is an advanced topic and is covered in the [Log Path overrides](https://splunk.github.io/splunk-connect-for-syslog/main/configuration/) section of the Configuration topic. - -#### Configure source filtering by source IP address or host name - -You must configure legacy and non-standard-compliant sources by source IP or hostname as included in the event. To identify sources that require this step, see ["Sources"](../sources/index.md). - -#### Configure compliance index/metadata overrides - -In some cases, devices that have been properly sourcetyped need to be further categorized by compliance, geography, or other criterion. You use the files `compliance_meta_by_source.conf` and `compliance_meta_by_source.csv` to do this. The `conf` file specifies a filter to uniquely identify the messages that should be overridden, and the `csv` file -lists one or more metadata items that you can override based on the filter name. This is an advanced topic, see [Override index or metadata based on host, IP, or subnet]{https://splunk.github.io/splunk-connect-for-syslog/main/configuration/#override-index-or-metadata-based-on-host-ip-or-subnet-compliance-overrides). +| Container Runtime and Orchestration | Operating Systems | +|-------------------------------------------------------------------|-------------------------------------------------------------------------------------| +| [MicroK8s](k8s-microk8s.md) | Ubuntu with Microk8s | +| [Podman 1.7 & 1.9 + systemd](podman-systemd-general.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | +| [Docker CE 19 (and greater) + systemd](docker-systemd-general.md) | RHEL or CentOS >7.7 (best option), Debian or Ubuntu 18.04LTS | +| [Docker Desktop + Compose](docker-compose-MacOS.md) | MacOS | +| [Docker Desktop + Compose](docker-compose.md) | RHEL or CentOS 8.1 & 8.2 (best option) | +| [Bring your own Environment](byoe-rhel8.md) | RHEL or CentOS 8.1 & 8.2 (best option) | +| [Offline Container Installation](docker-podman-offline.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | +| [Ansible+Docker Swarm](ansible-docker-swarm.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | +| [Ansible+Podman](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 20.10LTS(and higher) | +| [Ansible+Docker](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 18.04LTS(and higher) | \ No newline at end of file diff --git a/docs/gettingstarted/index.md b/docs/gettingstarted/index.md index ffaaeea6f1..6c29908e04 100644 --- a/docs/gettingstarted/index.md +++ b/docs/gettingstarted/index.md @@ -22,6 +22,6 @@ or resource constraint can cause data to be lost in transmission. ## Implementation +### [Quickstart Guide](quickstart_guide.md) ### [Splunk Setup](getting-started-splunk-setup.md) - ### [Runtime configuration](getting-started-runtime-configuration.md) diff --git a/docs/gettingstarted/quickstart_guide.md b/docs/gettingstarted/quickstart_guide.md index e85491f7c6..14c2f68cf8 100644 --- a/docs/gettingstarted/quickstart_guide.md +++ b/docs/gettingstarted/quickstart_guide.md @@ -1,4 +1,5 @@ # Quickstart Guide +This guide will enable you to quickly implement basic changes to your Splunk instance and set up a simple SC4S installation. It's a great starting point for working with SC4S and establishing a minimal operational solution. The same steps are thoroughly described in the [Splunk Setup](getting-started-splunk-setup.md) and [Runtime configuration](getting-started-runtime-configuration.md) sections. ### Splunk setup - Create the following default indexes that are used by SC4S diff --git a/mkdocs.yml b/mkdocs.yml index 9f26aae99d..eba307249a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,9 +30,9 @@ nav: - Load Balancers: "lb.md" - Getting Started: - Read First: "gettingstarted/index.md" + - Quickstart Guide: "gettingstarted/quickstart_guide.md" - Splunk Setup: "gettingstarted/getting-started-splunk-setup.md" - Runtime Configuration: "gettingstarted/getting-started-runtime-configuration.md" - - Quickstart Guide: "gettingstarted/quickstart_guide.md" - Select Runtime: - Podman + systemd: "gettingstarted/podman-systemd-general.md" - Docker CE + systemd: "gettingstarted/docker-systemd-general.md" @@ -45,7 +45,7 @@ nav: - Docker Swarm: "gettingstarted/ansible-docker-swarm.md" - Podman/Docker: "gettingstarted/ansible-docker-podman.md" - mk8s: "gettingstarted/ansible-mk8s.md" - - Create a parser: "gettingstarted/create-parser.md" + - Create a parser: "create-parser.md" - Configuration: "configuration.md" - Development: "developing/index.md" - Destinations: "destinations.md" From 685db5821be217243fda6eebbd7c88e5b34e9e53 Mon Sep 17 00:00:00 2001 From: JENNIFER WORTHINGTON <37708067+jenworthington@users.noreply.github.com> Date: Sun, 5 May 2024 22:00:23 -0700 Subject: [PATCH 3/9] Update getting-started-runtime-configuration.md --- .../getting-started-runtime-configuration.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index a8a8fc67fa..bfb1aab52d 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -1,11 +1,11 @@ # Implement a Container Runtime and SC4S -## OS-level configuration +## Step 1: Configure your OS to work with SC4S ### Tune your receiver buffer -The host linux OS receiver buffer size must be tuned to match the SC4S default. This helps to avoid event dropping at the network level. +The host Linux OS receiver buffer size must be tuned to match the SC4S default. This helps to avoid event dropping at the network level. The default receiver buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to -match this: +match your buffer: 1. Edit `/etc/sysctl.conf` using the following whole-byte values corresponding to 16 MB: ```bash @@ -18,10 +18,8 @@ net.core.rmem_max = 17039360 sysctl -p ``` -3. Verify that the kernel does not drop packets by periodically monitoring the buffer using the command -`netstat -su | grep "receive errors"`. - -* Failure to tune the kernel for high-volume traffic results in message loss, which can be +3. To verify that the kernel does not drop packets, periodically monitor the buffer using the command +`netstat -su | grep "receive errors"`. Failure to tune the kernel for high-volume traffic results in message loss, which can be unpredictable and difficult to detect. The default values for receiver kernel buffers in most distributions is 2 MB, which may not be adequate for your configuration. @@ -33,7 +31,6 @@ IPv4 forwarding must be enabled for container networking. ```sudo sysctl net.ipv4.ip_forward``` * To enable IPv4 forwarding: ```sudo sysctl net.ipv4.ip_forward=1``` - * To ensure your changes persist upon reboot: * Define sysctl settings through files in ```/usr/lib/sysctl.d/```, ```/run/sysctl.d/```, and ```/etc/sysctl.d/```. * To override only specific settings, either add a file with a lexically later name in ```/etc/sysctl.d/``` and put following setting there or find this specific setting in one of the existing configuration files and set the value to ```1```. @@ -42,7 +39,7 @@ IPv4 forwarding must be enabled for container networking. net.ipv4.ip_forward=1 ``` -## Create local directory structure +## Step 2: Create your local directory structure Create the following three directories: @@ -60,7 +57,7 @@ topic for information about the directory structure that the archive uses. When you create these directories, make sure that they match the volume mounts specified in the sc4s.service [unit file](podman-systemd-general.md#unit-file). Failure to do this will cause SC4S to abort at startup. -## Select a Container Runtime and SC4S Configuration +## Step 3: Select a Container Runtime and SC4S Configuration | Container Runtime and Orchestration | Operating Systems | |-------------------------------------------------------------------|-------------------------------------------------------------------------------------| @@ -73,4 +70,4 @@ sc4s.service [unit file](podman-systemd-general.md#unit-file). Failure to do thi | [Offline Container Installation](docker-podman-offline.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | | [Ansible+Docker Swarm](ansible-docker-swarm.md) | RHEL 8.x or CentOS 8.x (best option), Debian or Ubuntu 18.04LTS | | [Ansible+Podman](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 20.10LTS(and higher) | -| [Ansible+Docker](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 18.04LTS(and higher) | \ No newline at end of file +| [Ansible+Docker](ansible-docker-swarm.md) | RHEL 7.x/8.x or CentOS 7.x/8.x (best option), Debian or Ubuntu 18.04LTS(and higher) | From 1f186320574ab718a1b76dd17314d9431c03bbd1 Mon Sep 17 00:00:00 2001 From: JENNIFER WORTHINGTON <37708067+jenworthington@users.noreply.github.com> Date: Fri, 10 May 2024 12:34:14 -0700 Subject: [PATCH 4/9] Update getting-started-runtime-configuration.md --- docs/gettingstarted/getting-started-runtime-configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index bfb1aab52d..47edb4ddef 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -3,7 +3,7 @@ ## Step 1: Configure your OS to work with SC4S ### Tune your receiver buffer -The host Linux OS receiver buffer size must be tuned to match the SC4S default. This helps to avoid event dropping at the network level. +You must tune the host Linux OS receiver buffer size to match the SC4S default. This helps to avoid event dropping at the network level. The default receiver buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to match your buffer: From 3c51f7eca9159301497d984d4f59bca4127ea8d7 Mon Sep 17 00:00:00 2001 From: JENNIFER WORTHINGTON <37708067+jenworthington@users.noreply.github.com> Date: Sat, 1 Jun 2024 23:44:55 -0700 Subject: [PATCH 5/9] Update getting-started-runtime-configuration.md final pass, suggestions incorporated --- .../gettingstarted/getting-started-runtime-configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index 47edb4ddef..24ba24fb7b 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -2,9 +2,9 @@ # Implement a Container Runtime and SC4S ## Step 1: Configure your OS to work with SC4S -### Tune your receiver buffer +### Tune your receive buffer You must tune the host Linux OS receiver buffer size to match the SC4S default. This helps to avoid event dropping at the network level. -The default receiver buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to +The default receive buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to match your buffer: 1. Edit `/etc/sysctl.conf` using the following whole-byte values corresponding to 16 MB: @@ -20,7 +20,7 @@ sysctl -p 3. To verify that the kernel does not drop packets, periodically monitor the buffer using the command `netstat -su | grep "receive errors"`. Failure to tune the kernel for high-volume traffic results in message loss, which can be -unpredictable and difficult to detect. The default values for receiver kernel buffers in most distributions is 2 MB, +unpredictable and difficult to detect. The default values for receive kernel buffers in most distributions is 2 MB, which may not be adequate for your configuration. ### Configure IPv4 forwarding From 4b63998be95dbc6f1711e5656c562fda96943a9a Mon Sep 17 00:00:00 2001 From: JENNIFER WORTHINGTON <37708067+jenworthington@users.noreply.github.com> Date: Sat, 1 Jun 2024 23:50:58 -0700 Subject: [PATCH 6/9] Update quickstart_guide.md comments incorporated --- docs/gettingstarted/quickstart_guide.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/gettingstarted/quickstart_guide.md b/docs/gettingstarted/quickstart_guide.md index 81c2099173..b3474ba227 100644 --- a/docs/gettingstarted/quickstart_guide.md +++ b/docs/gettingstarted/quickstart_guide.md @@ -1,4 +1,5 @@ # Quickstart Guide + This guide will enable you to quickly implement basic changes to your Splunk instance and set up a simple SC4S installation. It's a great starting point for working with SC4S and establishing a minimal operational solution. The same steps are thoroughly described in the [Splunk Setup](getting-started-splunk-setup.md) and [Runtime configuration](getting-started-runtime-configuration.md) sections. ### Splunk setup From d46f76d3a831ca46b92a607cc91ed43674811996 Mon Sep 17 00:00:00 2001 From: mstopa-splunk <139441697+mstopa-splunk@users.noreply.github.com> Date: Wed, 3 Jul 2024 15:10:33 +0200 Subject: [PATCH 7/9] Update getting-started-runtime-configuration.md --- docs/gettingstarted/getting-started-runtime-configuration.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index 24ba24fb7b..b7b45a677a 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -3,7 +3,7 @@ ## Step 1: Configure your OS to work with SC4S ### Tune your receive buffer -You must tune the host Linux OS receiver buffer size to match the SC4S default. This helps to avoid event dropping at the network level. +You must tune the host Linux OS receive buffer size to match the SC4S default. This helps to avoid event dropping at the network level. The default receive buffer for SC4S is 16 MB for UDP traffic, which should be acceptable for most environments. To set the host OS kernel to match your buffer: @@ -26,7 +26,8 @@ which may not be adequate for your configuration. ### Configure IPv4 forwarding In many distributions, for example CentOS provisioned in AWS, IPv4 forwarding is not enabled by default. -IPv4 forwarding must be enabled for container networking. +IPv4 forwarding must be enabled for container networking. + * To check that IPv4 forwarding is enabled: ```sudo sysctl net.ipv4.ip_forward``` * To enable IPv4 forwarding: From 8ab7982b206fe2735034c742cb229625fa40d1d9 Mon Sep 17 00:00:00 2001 From: mstopa-splunk <139441697+mstopa-splunk@users.noreply.github.com> Date: Wed, 3 Jul 2024 15:16:13 +0200 Subject: [PATCH 8/9] Update getting-started-runtime-configuration.md --- docs/gettingstarted/getting-started-runtime-configuration.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index b7b45a677a..1f02a2ff67 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -30,8 +30,10 @@ IPv4 forwarding must be enabled for container networking. * To check that IPv4 forwarding is enabled: ```sudo sysctl net.ipv4.ip_forward``` + * To enable IPv4 forwarding: ```sudo sysctl net.ipv4.ip_forward=1``` + * To ensure your changes persist upon reboot: * Define sysctl settings through files in ```/usr/lib/sysctl.d/```, ```/run/sysctl.d/```, and ```/etc/sysctl.d/```. * To override only specific settings, either add a file with a lexically later name in ```/etc/sysctl.d/``` and put following setting there or find this specific setting in one of the existing configuration files and set the value to ```1```. From ec805df91c01bee508dcb4f97ba07ef2e390c086 Mon Sep 17 00:00:00 2001 From: mstopa-splunk <139441697+mstopa-splunk@users.noreply.github.com> Date: Wed, 3 Jul 2024 15:27:21 +0200 Subject: [PATCH 9/9] Update getting-started-runtime-configuration.md --- .../getting-started-runtime-configuration.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/gettingstarted/getting-started-runtime-configuration.md b/docs/gettingstarted/getting-started-runtime-configuration.md index 1f02a2ff67..82651a2129 100644 --- a/docs/gettingstarted/getting-started-runtime-configuration.md +++ b/docs/gettingstarted/getting-started-runtime-configuration.md @@ -28,12 +28,8 @@ which may not be adequate for your configuration. In many distributions, for example CentOS provisioned in AWS, IPv4 forwarding is not enabled by default. IPv4 forwarding must be enabled for container networking. -* To check that IPv4 forwarding is enabled: -```sudo sysctl net.ipv4.ip_forward``` - -* To enable IPv4 forwarding: -```sudo sysctl net.ipv4.ip_forward=1``` - +* To check that IPv4 forwarding is enabled: ```sudo sysctl net.ipv4.ip_forward``` +* To enable IPv4 forwarding: ```sudo sysctl net.ipv4.ip_forward=1``` * To ensure your changes persist upon reboot: * Define sysctl settings through files in ```/usr/lib/sysctl.d/```, ```/run/sysctl.d/```, and ```/etc/sysctl.d/```. * To override only specific settings, either add a file with a lexically later name in ```/etc/sysctl.d/``` and put following setting there or find this specific setting in one of the existing configuration files and set the value to ```1```.