[FR] Skip eql optimizations on parsing query for unique fields

[Mikaayenson]

Issues

https://github.com/elastic/ia-trade-team/issues/218

Summary

At least one rule (name = "Linux init (PID 1) Secret Dump via GDB") was not populating related fields because it was being optimized incorrectly. This issue is related to #2611 which involves a bug in EQL to properly optimize queries in inset.

I've passed eql.parser.skip_optimizations to align with our test_all_rule_queries_optimized unit test which also does not optimize.

Testing

• Unit tests should still pass
• Run view-rule on the rule above to show required_fields populated

Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "This rule monitors for the potential memory dump of the init process (PID 1) through gdb. Attackers may leverage memory dumping techniques to attempt secret extraction from privileged processes. Tools that display this behavior include \"truffleproc\" and \"bash-memory-dump\". This behavior should not happen by default, and should be investigated thoroughly.",
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "eql",
  "license": "Elastic License v2",
  "name": "Linux init (PID 1) Secret Dump via GDB",
  "query": "process where host.os.type == \"linux\" and event.action == \"exec\" and event.type == \"start\" and \nprocess.name == \"gdb\" and process.args in (\"--pid\", \"-p\") and process.args == \"1\"\n",
  "references": [
    "https://github.com/controlplaneio/truffleproc",
    "https://github.com/hajzer/bash-memory-dump"
  ],
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "host.os.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.args",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],
  "risk_score": 47,
  "rule_id": "d4ff2f53-c802-4d2e-9fb9-9ecc08356c3f",
  "setup": "\nThis rule requires data coming in from Elastic Defend.\n\n### Elastic Defend Integration Setup\nElastic Defend is integrated into the Elastic Agent using Fleet. Upon configuration, the integration allows the Elastic Agent to monitor events on your host and send data to the Elastic Security app.\n\n#### Prerequisite Requirements:\n- Fleet is required for Elastic Defend.\n- To configure Fleet Server refer to the [documentation](https://www.elastic.co/guide/en/fleet/current/fleet-server.html).\n\n#### The following steps should be executed in order to add the Elastic Defend integration on a Linux System:\n- Go to the Kibana home page and click \"Add integrations\".\n- In the query bar, search for \"Elastic Defend\" and select the integration to see more details about it.\n- Click \"Add Elastic Defend\".\n- Configure the integration name and optionally add a description.\n- Select the type of environment you want to protect, either \"Traditional Endpoints\" or \"Cloud Workloads\".\n- Select a configuration preset. Each preset comes with different default settings for Elastic Agent, you can further customize these later by configuring the Elastic Defend integration policy. [Helper guide](https://www.elastic.co/guide/en/security/current/configure-endpoint-integration-policy.html).\n- We suggest selecting \"Complete EDR (Endpoint Detection and Response)\" as a configuration setting, that provides \"All events; all preventions\"\n- Enter a name for the agent policy in \"New agent policy name\". If other agent policies already exist, you can click the \"Existing hosts\" tab and select an existing policy instead.\nFor more details on Elastic Agent configuration settings, refer to the [helper guide](https://www.elastic.co/guide/en/fleet/8.10/agent-policy.html).\n- Click \"Save and Continue\".\n- To complete the integration, select \"Add Elastic Agent to your hosts\" and continue to the next section to install the Elastic Agent on your hosts.\nFor more details on Elastic Defend refer to the [helper guide](https://www.elastic.co/guide/en/security/current/install-endpoint.html).\n\n",
  "severity": "medium",
  "tags": [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Credential Access",
    "Data Source: Elastic Defend"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0006",
        "name": "Credential Access",
        "reference": "https://attack.mitre.org/tactics/TA0006/"
      },
      "technique": [
        {
          "id": "T1003",
          "name": "OS Credential Dumping",
          "reference": "https://attack.mitre.org/techniques/T1003/",
          "subtechnique": [
            {
              "id": "T1003.007",
              "name": "Proc Filesystem",
              "reference": "https://attack.mitre.org/techniques/T1003/007/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "eql",
  "version": 4
}

[eric-forte-elastic]

Similar to other optimization fixes. Manual review, LGTM 👍

🟢 Testing Results, required fields present

"required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "host.os.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.args",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],

Details

… 218-add-required_fields-to-detections-rules-that-are-missing-those [!] is  v0.1.0 via  v3.8.18 (venv) on  eric.forte 
❯ python -m detection_rules view-rule /home/forteea1/Code/clean_mains/detection-rules/rules/linux/credential_access_gdb_init_memory_dump.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "This rule monitors for the potential memory dump of the init process (PID 1) through gdb. Attackers may leverage memory dumping techniques to attempt secret extraction from privileged processes. Tools that display this behavior include \"truffleproc\" and \"bash-memory-dump\". This behavior should not happen by default, and should be investigated thoroughly.",
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "eql",
  "license": "Elastic License v2",
  "name": "Linux init (PID 1) Secret Dump via GDB",
  "query": "process where host.os.type == \"linux\" and event.action == \"exec\" and event.type == \"start\" and \nprocess.name == \"gdb\" and process.args in (\"--pid\", \"-p\") and process.args == \"1\"\n",
  "references": [
    "https://github.com/controlplaneio/truffleproc",
    "https://github.com/hajzer/bash-memory-dump"
  ],
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "host.os.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.args",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],
  "risk_score": 47,
  "rule_id": "d4ff2f53-c802-4d2e-9fb9-9ecc08356c3f",
  "setup": "\nThis rule requires data coming in from Elastic Defend.\n\n### Elastic Defend Integration Setup\nElastic Defend is integrated into the Elastic Agent using Fleet. Upon configuration, the integration allows the Elastic Agent to monitor events on your host and send data to the Elastic Security app.\n\n#### Prerequisite Requirements:\n- Fleet is required for Elastic Defend.\n- To configure Fleet Server refer to the [documentation](https://www.elastic.co/guide/en/fleet/current/fleet-server.html).\n\n#### The following steps should be executed in order to add the Elastic Defend integration on a Linux System:\n- Go to the Kibana home page and click \"Add integrations\".\n- In the query bar, search for \"Elastic Defend\" and select the integration to see more details about it.\n- Click \"Add Elastic Defend\".\n- Configure the integration name and optionally add a description.\n- Select the type of environment you want to protect, either \"Traditional Endpoints\" or \"Cloud Workloads\".\n- Select a configuration preset. Each preset comes with different default settings for Elastic Agent, you can further customize these later by configuring the Elastic Defend integration policy. [Helper guide](https://www.elastic.co/guide/en/security/current/configure-endpoint-integration-policy.html).\n- We suggest selecting \"Complete EDR (Endpoint Detection and Response)\" as a configuration setting, that provides \"All events; all preventions\"\n- Enter a name for the agent policy in \"New agent policy name\". If other agent policies already exist, you can click the \"Existing hosts\" tab and select an existing policy instead.\nFor more details on Elastic Agent configuration settings, refer to the [helper guide](https://www.elastic.co/guide/en/fleet/8.10/agent-policy.html).\n- Click \"Save and Continue\".\n- To complete the integration, select \"Add Elastic Agent to your hosts\" and continue to the next section to install the Elastic Agent on your hosts.\nFor more details on Elastic Defend refer to the [helper guide](https://www.elastic.co/guide/en/security/current/install-endpoint.html).\n\n",
  "severity": "medium",
  "tags": [
    "Domain: Endpoint",
    "OS: Linux",
    "Use Case: Threat Detection",
    "Tactic: Credential Access",
    "Data Source: Elastic Defend"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0006",
        "name": "Credential Access",
        "reference": "https://attack.mitre.org/tactics/TA0006/"
      },
      "technique": [
        {
          "id": "T1003",
          "name": "OS Credential Dumping",
          "reference": "https://attack.mitre.org/techniques/T1003/",
          "subtechnique": [
            {
              "id": "T1003.007",
              "name": "Proc Filesystem",
              "reference": "https://attack.mitre.org/techniques/T1003/007/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "eql",
  "version": 4
}