[PAD] Add new advanced analytical security integration Privileged Access Detection

[sodhikirti07]

Proposed commit message

Added new advanced analytical package to detect Privileged access in your environment.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Added changelog, manifest and README. Make sure README has all the instructions to install this package and related components.
• The pad-ml.json file has no typos. Windows and Linux jobs are excluding background processes. In total, there are 21 ML jobs (9 windows, 9 okta and 3 linux)
• Transform and ingest pipelines are working and collecting data as expected.
• Dashboards are installed and working

How to test this PR locally

Related issues

Screenshots

[sodhikirti07]

Screencaps:

Package overview:

Transforms are healthy and working:

Ingest pipeline is working and generating data:

Dashboards are working:

Anomaly detection s working:

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9b864a0

History

• 💚 Build #22981 succeeded 072207f
• 💔 Build #22654 failed e5d30c5

[elastic-sonarqube]

Quality Gate failed

Failed conditions
8.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[susan-shu-c]

I feel that even if this is an experimental release it's still maybe worth of 0.5.0?

[sodhikirti07]

@susan-shu-c Do you know what difference would it make? Also, we've always shipped experimental as 0.0.1, so I'm a bit reluctant.

[mgarzon]

minor thing, but can we check if inputString is not null before iterating.
if (inputString != null) { ....

[sodhikirti07]

@mgarzon The if condition on line 20 is checking for the same.

[jmcarlock]

I did one more test of the readme instructions and install on 9.0.0-SNAPSHOT locally, with test data from the applied ML cluster. The only thing I was not able to check were the component pipeline mappings, as I'm not sure how to test them outside of connecting Elastic Defend. I'm including my full test process.

First I installed the PAD integration.

Then I verified that the document I was using to test could be processed with the pipeline:

GET _ingest/pipeline/0.0.1-ml_pad_ingest_pipeline/_simulate?verbose=true
{
  "docs": [
    {
      "_source": {
        "event": {
          "category":
            "process"
        },
        "host": {
          "os": {
            "type":
              "linux"
          }
        },
        "process": {
          "command_line": "testentropy"
        }
      }
    }
  ]
}

This output a value for command_line_entropy:

              "process.command_line_entropy": 2.8453509366224368

Then I added a custom ingest pipeline (and also verified this produced the same results through the GUI):

PUT _ingest/pipeline/logs-endpoint.events.process@custom
{
  "processors": [
    {
      "pipeline": {
        "name": "0.0.1-ml_pad_ingest_pipeline",
        "ignore_missing_pipeline": true,
        "ignore_failure": true
      }
    }
  ]
}

And a component template (same here, verified the same results through the GUI):

PUT _component_template/logs-endpoint.events.process-default@custom
{
  "template": {
    "settings": {
      "index": {
        "default_pipeline": "0.0.1-ml_pad_ingest_pipeline"
      }
    },
    "mappings": {
      "properties": {
        "process": {
          "type": "object",
          "properties": {
            "command_line_entropy": {
              "type": "double"
            }
          }
        }
      }
    }
  }
}

Then I rollover the data stream to create a new index:

POST logs-endpoint.events.process-default/_rollover

The index then rolls over:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-endpoint.events.process-default-2025.03.10-000020",
  "new_index": ".ds-logs-endpoint.events.process-default-2025.03.10-000021",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {}
}

Finally when I put my document into the data stream:

POST logs-endpoint.events.process-default/_doc
{
  "event": {
    "category":
      "process"
  },
  "host": {
    "os": {
      "type":
        "linux"
    }
  },
  "process": {
    "command_line": "testentropy"
  }
}

I get an "all good":

{
  "_index": ".ds-logs-endpoint.events.process-default-2025.03.10-000021",
  "_id": "AZWBthmFPjYSSlRNzx9z",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

But when I query the index or the data stream, I do not see a value for command_line_entropy:

GET logs-endpoint.events.process-default/_search

      {
        "_index": ".ds-logs-endpoint.events.process-default-2025.03.10-000021",
        "_id": "AZWBthmFPjYSSlRNzx9z",
        "_score": 1,
        "_source": {
          "@timestamp": "2025-03-10T20:19:23.039Z",
          "event": {
            "category": "process"
          },
          "host": {
            "os": {
              "type": "linux"
            }
          },
          "process": {
            "command_line": "testentropy"
          }
        }
      }

[sodhikirti07]

@jmcarlock Thanks, Gus, for thoroughly testing the ingest pipeline. I'll run this step again on my end to investigate what's happening.

[jmcarlock]

@sodhikirti07 I think it's just an index issue since I am not enrolling an agent using Elastic Defend. Thanks for taking this up!

Also, for some reason I wasn't able to get the mappings to work either. Here was the test command.

GET logs-endpoint.events.process-default/_mapping/field/process.command_line_entropy

I had this issue testing with 8.17.2 and 9.0.0-SNAPSHOT. With both I do not have the default component template, which I think is only created when an agent is first enrolled and starts sending data.

[sodhikirti07]

@jmcarlock I followed the above steps and can confirm that I can see the process.command_line_entropy field under @mgarzon cluster. Note that this cluster has a Linux agent installed which is in active state. See below screenshots.

The custom index and component template are configured:

The process.command_line_entropy field is collecting data:

[sodhikirti07]

@elastic/ml-ui Can anyone from the ML UI team take a look at this PR? This is the initial release of our new security capability to detect privileged access.

[qn895]

ML changes LGTM 🎉

[elastic-vault-github-plugin-prod]

Package pad - 0.0.1 containing this change is available at https://epr.elastic.co/package/pad/0.0.1/