From 889d78c79a1fb2f8d5dd25aa6c723cd8d021fb09 Mon Sep 17 00:00:00 2001 From: JENNIFER WORTHINGTON <37708067+jenworthington@users.noreply.github.com> Date: Mon, 3 Jun 2024 03:29:47 -0700 Subject: [PATCH] docs: update getting-started-splunk-setup.md (#2417) --- .../getting-started-splunk-setup.md | 105 ++++++++++-------- 1 file changed, 59 insertions(+), 46 deletions(-) diff --git a/docs/gettingstarted/getting-started-splunk-setup.md b/docs/gettingstarted/getting-started-splunk-setup.md index 57d4730a33..5508727af3 100644 --- a/docs/gettingstarted/getting-started-splunk-setup.md +++ b/docs/gettingstarted/getting-started-splunk-setup.md @@ -1,49 +1,62 @@ # Splunk setup -## Create Indexes - -SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when -using the SC4S defaults. SC4S can be easily customized to use different indexes if desired. - -* email -* epav -* epintel -* infraops -* netauth -* netdlp -* netdns -* netfw -* netids -* netlb -* netops -* netwaf -* netproxy -* netipam -* oswin -* oswinsec -* osnix -* print -* _metrics (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index) - -## Configure the Splunk HTTP Event Collector - -- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky -session. Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native syslog-ng load balancing) if no load balancer is in -place. In most scenarios the recommendation is to use an external load balancer, as that makes longer term -maintenance simpler by eliminating the need to manually keep the list of HEC URLs specified in sc4s current. However, if a LB is not -available, native load balancing can be used with 10 or fewer Indexers where HEC is used exclusively for syslog. - - In either case, it is _strongly_ recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than -an intermediate tier of HWFs. -- Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, _metrics, and all indexes used as -event destinations. - -* NOTE: It is recommended that the "Selected Indexes" on the token configuration page be left blank so that the token has access to -_all_ indexes, including the `lastChanceIndex`. If this list is populated, extreme care must be taken to keep it up to date, as an attempt to -send data to an index not in this list will result in a `400` error from the HEC endpoint. Furthermore, the `lastChanceIndex` will _not_ be -consulted in the event the index specified in the event is not configured on Splunk. Keep in mind just _one_ bad message will "taint" the -whole batch (by default 1000 events) and prevent the entire batch from being sent to Splunk. -* In case you are not using TLS on SC4S- turn off SSL on global settings for HEC in Splunk. -- Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud) -or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your +To ensure proper integration for SC4S and Splunk, perform the following tasks in your Splunk instance: + +1. Create your SC4S indexes in Splunk. +2. Configure your HTTP event collector. + + +## Step 1: Create indexes within Splunk + +SC4S maps each sourcetype to the following indexes by default. You will also need to create these indexes in Splunk: + +* `email` +* `epav` +* `epintel` +* `fireeye` +* `gitops` +* `infraops` +* `netauth` +* `netdlp` +* `netdns` +* `netfw` +* `netids` +* `netlb` +* `netops` +* `netwaf` +* `netproxy` +* `netipam` +* `oswin` +* `oswinsec` +* `osnix` +* `print` +* `_metrics` (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index) + +If you use custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information. + +## Step 2: Configure your HTTP event collector + +See [Use the HTTP event collector](https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/UsetheHTTPEventCollector) for HEC configuration instructions based on your Splunk type. +Keep in mind the following best practices specific to HEC for SC4S: + +* Make sure that the HEC token created for SC4S has permissions to write to `_metrics` and all event destination indexes. +* You can leave "Selected Indexes" blank on the token configuration page so that the token has access to +all indexes, including the `lastChanceIndex`. If you do populate this field, take extreme care to keep it up to date; an attempt to +send data to an index that is not in this list results in a `400` error from the HEC endpoint. The `lastChanceIndex` will not be +consulted if the index specified in the event is not configured on Splunk and the entire batch is then not sent to Splunk. +* SC4S traffic should be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of heavy forwarders. +* SC4S traffic must be sent to HEC endpoints that are configured directly on the indexers. + +### Create a load balancing mechanism +In some configurations, you should ensure output balancing from SC4S to Splunk indexers. To do this, you create a load balancing mechanism between SC4S and Splunk indexers. Note that this should not be confused with load balancing between [sources and SC4S](../lb.md). + +When configuring your load balancing mechanism, keep in mind the following: + +* Splunk Cloud provides an internal ELB on TCP 443. +* For Splunk Enterprise set up your Splunk HTTP Event Collector with the HEC endpoints behind a load balancer. +* An external load balancer simplifies long-term maintenance by eliminating the need to manually keep the list of HEC URLs specified in SC4S current. Set up a load balancer using virtual IP and configured for https round-robin without sticky session. +* If a load balancer is not available, you can configure a list of HEC endpoint URLs with native syslog-ng load balancing. For internal load balancing of syslog-ng you should: + * Load balance ten or fewer indexers. + * Use HEC exclusively for syslog. + * Have SC4S extract timestamps from messages (default behavior) rather than use the time of receipt for the message.