setup_splunk.md

Splunk

For the lastest installation instructions for this repo please see this blog post: IR TALES: THE QUEST FOR THE HOLY SIEM: SPLUNK + SYSMON + OSQUERY + ZEEK

Init Ansible playbook

1. vim macos-workshop/ChooseYourSIEMAdventure/hosts.ini and add the Graylog server IP address under [splunk]
2. vim macos-workshop/ChooseYourSIEMAdventure/group_vars/all.yml and set:
   1. base_domain – <external_domain - group_vars/all.yml>
   2. timezone – OPTIONAL – The default timezone is UTC+0
   3. siem_username – Ignore this setting
   4. siem_password – Set the Splunk admin password
   5. 
3. vim macos-workshop/ChooseYourSIEMAdventure/group_vars/splunk.yml and set:
   1. hostname – Set the desired hostname for the server
   2. splunk_version – Set the desired version of Splunk to use
   3. splunk_dl_url – Set to the URL to download Splunk
   4. beats_port – OPTIONAL – Set the port to ingest logs using BEATs clients
   5. elastic_version – OPTIONAL – Set the desired version of Logstash to use with Splunk – best to leave as default
      1. elastic_repo_version – Change the repo version to install Logstash
   6. 

Run Ansible playbook

1. ansible-playbook -i hosts.ini deploy_splunk.yml -u ubuntu --key-file terraform/ssh_keys/id_rsa
   1. 

Generate Let's Encrypt certificate

1. SSH into Elastic EC2 instance
2. sudo su
3. apt install certbot python3-certbot-nginx -y
4. certbot --nginx -d splunk.<external domain>
5. Enter the e-mail for the admin of the domain
6. Enter A for Terms of Service
7. Enter N to share e-mail with EFF
8. Enter 2 to redirect HTTP traffic to HTTPS
9. Review NGINX config: /etc/nginx/conf.d/graylog.conf
10. systemctl restart nginx

Create workshop user

1. Open browser to https://splunk.<external domain> and login
2. Enter admin for username
3. Enter <siem_password> into Password
4. Settings > Users and Authentication > Users
5. Select "New User" in top right
6. Enter threathunter into name
7. Enter threathunter into full name
8. Leave e-mail blank
9. Enter a password for the user
10. UNcheck Require password change on first login
11. 
12. Select "Create user"

Create indexes

1. Settings > Data > Indexes
2. Select "New index" in the top right
3. Enter osquery into Index name
4. Select Events for Index Data Type
5. Ensure "Search & Reporting" is selected for App
6. 
7. Select "Save"
8. Repeat for the test index

Install Osquery Splunk TA

1. Apps > Find More Apps
2. Search for osquery
3. Select "Install" for add-on for Osquery
4. Enter Splunk account credentials
5. Install app
6. Restart Splunk

Install Sysmon Splunk TA

1. Apps > Find More Apps
2. Search for Sysmon
3. Select "Install" for Splunk Add-On for Microsoft Sysmon
4. Enter Splunk account credentials
5. Install app
6. Restart Splunk

Create HEC tokens

1. Settings > Data > Data inputs
2. Type > HTTP Event Collector
3. Select "New Token" in the top right
4. Select Source
5. Enter osquery-hec-token into Name
6. 
7. Select "Next"
8. Input Settings
9. Select osquery:results for sourcetype
10. Select osquery for allowed indexes-
11. 
12. Select "Next"
13. Review
14. Select "Next"
15. Done
16. Repeat for test index
17. Settings > Data > Data inputs
18. Type > HTTP Event Collector
19. Copy HEC tokens

Setup Logstash with HEC tokens

1. SSH into Splunk EC2 instance
2. sudo su
3. systemctl stop logstash
4. rm /etc/logstash/conf.d/3*-output-*.conf
5. Delete all the outputs
6. curl https://raw.githubusercontent.com/OTRF/macos-workshops/main/conf/splunk/30-output-logstash-service.conf?token=AIRH4CATUWWQ5SJO44XIHGLA3UJTU --output 30-output-splunk-osquery.conf
7. sed -i "s#{{ logstash_type }}#osquery_json#g" 30-output-splunk-osquery.conf
8. Set Logstash type
9. sed -i "s#{{ hec_token }}#<osquery-hec-token>#g" 30-output-splunk-osquery.conf
10. Set Splunk HEC token
11. sed -i "s#{{ splunk_source_type }}#osquery:results#g" 30-output-splunk-osquery.conf
12. Set Splunk source type
13. 

DEBUGGING Logstash

• tail -f /var/log/logstash/logstash-plain

Clearing index data

1. SSH into Splunk
2. /opt/splunk/bin/splunk stop
3. /opt/splunk/bin/splunk clean eventdata -index <index name>

References

• How to Set up Splunk HEC the Smart Way
• What is the reseasoning behind the Sysmon sourcetype?
• How to create HEC token through an API call?
• Using the AUTH LOGIN sessionkey with Curl SplunkAPI
• Using Ansible uri module to add users to splunk via REST API
• Splunk REST API Reference Manual
• Remove indexes and indexed data