CONTRIBUTING.md

Contributing (Development) Guide

Since Splunk support Python 3.7 (deprecated as of June 2023), and cant be easily installed we use Python 3.8 to build app. In order to use python 3.8 we use Python Virtual environment.

Prerequisites

You need Python 3.8 for development and package building.

Python 3.8

• Install Python 3.8 brew install python@3.8
• Create venv with Python3.8: python3.8 -m venv venv
• Activate it: source venv/bin/activate
• Install dependencies for packing:

    # install tools for packing the project
    make dev-install-dependencies-pack
    # this installs slim, which wants to be installed into /usr/local/bin/slim
    sudo make dev-install-dependencies-pack-sudo
    # install library dependencies
    make dev-install-dependencies-lib
    # If you get "ImportError: failed to find libmagic.  Check your installation" error when
    # running inspect target, you also need to install libmagic dependency
    brew install libmagic
    # If you get "getopt: illegal option -- n" this means you don't have GNU getopt tool installed
    # You can install it using command below
    brew install gnu-getopt

• Test it:
  • By packing the application: make pack
  • By inspecting the application: make inspect
• Install dependencies for development:

    # install tools for development
    make dev-install-dependencies-for-development

• Install pre-commit:

  pre-commit install

  • If this fails, then:
    • Determine current python: which python - it should be something like - current_directory/venv/bin/python
    • Update following file: .git/hooks/pre-commit - where you replace INSTALL_PYTHON with the path to the Python from previous command:

      #!/usr/bin/env bash
      # File generated by pre-commit: https://pre-commit.com
      # ID: 138fd403232d2ddd5efb44317e38bf03
      
      # start templated
      INSTALL_PYTHON="$(cd "$(dirname "$0")" && pwd)"/../../venv/bin/python
      ARGS=(hook-impl --config=.pre-commit-config.yaml --hook-type=pre-commit)
      # end templated
      
      HERE="$(cd "$(dirname "$0")" && pwd)"
      ARGS+=(--hook-dir "$HERE" -- "$@")
      
      if [ -x "$INSTALL_PYTHON" ]; then
          exec "$INSTALL_PYTHON" -mpre_commit "${ARGS[@]}"
      elif command -v pre-commit > /dev/null; then
          exec pre-commit "${ARGS[@]}"
      else
          echo '`pre-commit` not found.  Did you forget to activate your virtualenv?' 1>&2
          exit 1
      fi

• Test it:
  • By checking all the source files: pre-commit run -a

Build App

• Use make target:

  make pack

• Run commands manually:

  ucc-gen --source TA_dataset --ta-version "$(jq -r '.meta.version' globalConfig.json)"
  slim package output/TA_dataset -o release

Run Docker Splunk locally (Mac M1 machines)

Since Splunk does not have Docker image for Apple Sillicon you may need to

• Use Docker Desktop 4.16.1 or newer and enable Use Virtualization framework in General and Rosetta in Features in Development, see more
• Run Splunk Docker image with --platform=linux/amd64 parameter, eg: docker run -it -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=Test0101 --platform=linux/amd64 --name splunk -p 8000:8000 splunk/splunk:latest start

To clean up container run docker container rm splunk command

Install Singularity Data Lake Add-On for Splunk to running Docker container

Assuming application was previously built

From existing release

• docker cp release/TA_dataset-2.0.3-SNAPSHOT.tar.gz splunk:/opt/splunk/etc/apps/
• docker exec splunk sudo tar -xvzf /opt/splunk/etc/apps/TA_dataset-2.0.3-SNAPSHOT.tar.gz -C /opt/splunk/etc/apps/
• docker exec splunk sudo chown -R splunk:splunk /opt/splunk/etc/apps/TA_dataset/
• docker exec splunk sudo -u splunk /opt/splunk/bin/splunk restart

Using mounted volume from built app

• Mount folder with built app docker run -it -v "$(pwd)/output/TA_dataset:/opt/splunk/etc/apps/TA_dataset/" -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=Test0101 --platform=linux/amd64 --name splunk -p 8000:8000 splunk/splunk:latest start

To apply changes build app again ucc-gen build --source TA_dataset

• Changes in python scripts take effect immediately without any restart
• Changes in static files like XML template take effect after restart docker exec splunk sudo -u splunk /opt/splunk/bin/splunk restart

Once application is installed create connection to DataSet environment under Configuration tab using Add button. Note that build cleans previously created configuration. To prevent removal of configuration while build

• copy configured configuration to home folder mkdir -p ~/splunk_dataset_app_configuration && cp -R ./output/TA_dataset/local/* ~/splunk_dataset_app_configuration/
• copy back to splunk mkdir -p ./output/TA_dataset/local/ && cp -R ~/splunk_dataset_app_configuration/* ./output/TA_dataset/local/

Alternative Build and Run Workflow for Docker using make commands

1. At the beginning of the day:
   1. Remove non-running container - make docker-splunk-remove
   2. Create package - make pack
   3. Run Splunk in Docker - make docker-splunk-run (if it already exists use make docker-splunk-start)
   4. You can combine this into - make docker-splunk-remove pack docker-splunk-run
2. Do your code changes (assuming docker is already running, see previous steps):
   1. Update source code - make dev-update-source

You have to do 1. when you are changing other files (matadata, assets, ...). If the container is still running, you can use make docker-splunk-kill.

Other Useful Commands

• Run Splunk without DataSet Add-On - make docker-splunk-run-vanilla
• Restart Splunk - make docker-splunk-restart
• Stop Splunk - make docker-splunk-stop
• Start stopped Splunk container - make docker-splunk-start
• Kill Splunk container - make docker-splunk-kill
• Remove Splunk container - make docker-splunk-remove
• Restore configuration - make dev-config-backup
• Backup configuration - make dev-config-restore - it's not clear whether it really works
• To see all available logs - make docker-splunk-list-logs
• To see particular log, you may use - make docker-splunk-tail-log LOG_NAME=log-file
  • Logs related to Splunk Python - make docker-splunk-tail-logs-python calls make docker-splunk-tail-log LOG_NAME=python.log
  • Logs related to Search command - make docker-splunk-tail-logs-app-search-command calls make docker-splunk-tail-log LOG_NAME="TA_dataset_search_command.log"

Where are errors:

• search_messages.log - error message that is shown in the UI, no stack trace :/

E2E Testing

We are using Playwright - https://playwright.dev/

Setup

• Install Node - brew install node
• Install modules - make e2e-install
• Create .env file based on .env.example with access keys

Run tests

• Make sure, that your Splunk is running - make docker-splunk-run
• Use make e2e-test - to run e2e tests without the browser
• Use make e2e-test-headed - to run e2e tests with the browser

Release

When code is merged, new tag is created and content of the release folder is updated with the tarball containing the latest version. You can release new version by following these instructions:

1. Go to Releases and remember what was the latest version released.
2. Click on Draft new release.
3. Create new release by:
   1. Pick the latest tag.
   2. As previous tag pick the tag of the latest release from 1.
   3. Click on Generate release notes.
   4. Upload the file from the release folder - TA_dataset-x.y.z.tar.gz.
   5. Append some short description to the Release title.
4. Click on Publish release.