docs: update getting-started-splunk-setup.md

[jenworthington]

I can't quite figure out how the customer is supposed to use this. It reads as steps but they don't seem like actual guided tasks. Is it more of a best practices topic or an overview? Or if it is numbered steps/process, then maybe we add links to each step? I'd love to get your thoughts and maybe we can discuss in our 1:1

[mstopa-splunk]

hi @jenworthington sure, that's what this section is about:

Topic: how to setup your Splunk instance to work with SC4S

Steps:

1. Create default indexes in Splunk
2. Set up the Splunk HTTP Event Collector

These are the two things that must be done to ensure SC4S-Splunk connection.

Ad 1 Indexes
You can use your custom set of indexes. But make sure that all of them, as well as the default set, are created in Splunk, else you will miss events processed by SC4S

Ad 2 HTTP event collector

• Refer to Splunk docs to see how to set it up
• But here are best practices to avoid problems:
  a. put HEC endpoints of your indexers behind a load balancer. Use native syslog-ng load balancing or, preferably, an external load balancer
  b. don't use an intermediate tier of HWFs
  c. make sure that the HEC token has permissions to write in the indexes that you'll need
  d. make sure that you either don't put any "Selected Indexes" or you carefully keep this list up to date
  e. If you're not using TLS on SC4S, turn it off in Splunk's HEC token too.

[mstopa-splunk]

partially solves #2358

[mstopa-splunk]

@jenworthington can you work on the new file docs/gettingstarted/getting-started-splunk-setup-new.md ? I will replace the old one with this one when we finish

[rjha-splunk]

Please check the comment.

[mstopa-splunk]

@rjha-splunk I left the file that you saw for reference for Jen, but please check docs/gettingstarted/getting-started-splunk-setup-new.md instead. It will replace the old one completely

[jenworthington]

Thanks for the new suggestions for structure, it was really helpful. I think I've captured all of the requested changes, take a look and let me know, happy to work on this one some more as needed.

[mstopa-splunk]

@jenworthington ready for the next iteration

[mstopa-splunk]

@jenworthington something went wrong and your changes to docs/gettingstarted/getting-started-splunk-setup.md from the last pass were not commited. I opened all previous comments again, please go through them and commit the final pass, I'm sorry for that situation

[mstopa-splunk]

@jenworthington ready for the final pass

[srv-rr-github-token]

🎉 This PR is included in version 3.27.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀