Skip to content

fix: removing outdated code in Kibana client auth #4495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Mar 24, 2025
Merged

fix: removing outdated code in Kibana client auth #4495

merged 6 commits into from
Mar 24, 2025

Conversation

traut
Copy link
Contributor

@traut traut commented Feb 25, 2025 •
edited
Loading

Pull Request

Issue link(s):

Motivation

Kibana API does not officially support username/password auth (docs), and has an explicit warning:

Access to internal Kibana API endpoints will be restricted in 9.0. Please move any integrations to publicly documented APIs.

Following the best practices, our goal is to drop username / password support for Kibana auth in our code and move to API key authentication.

Both cloud and standalone Kibana deployments support API key authentication.

Summary - What I changed

  • dropped support for username / password / cookies in Kibana client as insecure
  • adjust the code that uses Kibana class to use api_key

How To Test

  • unit tests pass

Checklist

  • Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
  • Added the meta:rapid-merge label if planning to merge within 24 hours
  • Secret and sensitive material has been managed correctly
  • Automated testing was updated or added to match the most common scenarios
  • Documentation and comments were added for features that require explanation

Contributor checklist

Sorry, something went wrong.

@traut traut added the bug Something isn't working label Feb 25, 2025
@github-actions GitHub Actions
Copy link
Contributor

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

  • Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
  • Include additional context or details about the problem.
  • Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

  • Code follows established design patterns within the repo and avoids duplication.
  • Code changes do not introduce new warnings or errors.
  • Variables and functions are well-named and descriptive.
  • Any unnecessary / commented-out code is removed.
  • Ensure that the code is modular and reusable where applicable.
  • Check for proper exception handling and messaging.

Testing

  • New unit tests have been added to cover the bug fix or edge cases.
  • Existing unit tests have been updated to reflect the changes.
  • Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
  • Validate that any rules affected by the bug are correctly updated.
  • Ensure that performance is not negatively impacted by the changes.
  • Verify that any release artifacts are properly generated and tested.

Additional Checks

  • Ensure that the bug fix does not break existing functionality.
  • Review the bug fix with a peer or team member for additional insights.
  • Verify that the bug fix works across all relevant environments (e.g., different OS versions).
  • Confirm that all dependencies are up-to-date and compatible with the changes.
  • Confirm that the proper version label is applied to the PR patch, minor, major.

Sorry, something went wrong.

@traut traut added the patch label Feb 25, 2025
@traut traut marked this pull request as ready for review February 25, 2025 13:50
@eric-forte-elastic
Copy link
Contributor

Just noting so that we do not forget, if we are dropping support for user/password we need to update our documentation in this repo and in https://github.com/elastic/DaC-Reference.

@botelastic botelastic bot added the python Internal python for the repository label Mar 5, 2025
@traut traut force-pushed the kibana-auth-fix branch from 162b778 to cf14883 Compare March 18, 2025 12:59
@traut traut mentioned this pull request Mar 18, 2025
Merged
@traut
Copy link
Contributor Author

traut commented Mar 18, 2025

related DaC-Reference change - elastic/DaC-Reference#25

@traut traut requested a review from Mikaayenson March 18, 2025 13:44
shashank-elastic
shashank-elastic approved these changes Mar 20, 2025
View reviewed changes
Copy link
Contributor

@shashank-elastic shashank-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from patch version update. The other changes look good

@eric-forte-elastic
Copy link
Contributor

Like @Mikaayenson suggested, I think this should be a major bump as this is a breaking change.

@eric-forte-elastic
Copy link
Contributor

Documentation for setting up API Key Auth

@Mikaayenson Mikaayenson added major and removed patch labels Mar 20, 2025
@traut traut force-pushed the kibana-auth-fix branch from b3c9f5e to 9b60e69 Compare March 24, 2025 10:59
@traut traut merged commit 65170c3 into main Mar 24, 2025
13 checks passed
@traut traut deleted the kibana-auth-fix branch March 24, 2025 11:28
r0ot added a commit to VigilantSec/detection-rules that referenced this pull request Apr 16, 2025
@r0ot @eric-forte-elastic
@sodhikirti07 @Samirbous @w0rk3r @shashank-elastic @Mpdreamz @Mikaayenson @terrancedejesus @reakaleek @traut @DefSecSentinel @BugOrFeature @github-actions @Aegrah @imays11 @frederikb96
Upstream merge 16 apr 25 (#24)
a7cd726 
* Temporaily Disable Changed FIles Workflow (elastic#4538)

* Temporaily Disable Changed FIles Workflow

* bump version

* Add new ML detection rules for Privileged Access Detection (elastic#4516)

Add detection-rules for privileged access detection integration

* Revert "Add new ML detection rules for Privileged Access Detection (elastic#4516)" (elastic#4548)

This reverts commit 2ff8d1b.

* Min stack rules from 4516 (elastic#4549)

* Update defense_evasion_posh_assembly_load.toml (elastic#4543)

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Change description and name of problemchild ML detection-rules (elastic#4545)

Changed description and name of problemchild ML detection-rules

* Prep for Release 9.0 (elastic#4550)

* [ci] Add new docs-builder automation. (elastic#4507)

* Add new docs automation

* Add path-pattern filters for documentation folders

* Update .github/workflows/docs-build.yml

Co-authored-by: Jan Calanog <nejcalanog@gmail.com>

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Jan Calanog <nejcalanog@gmail.com>
Co-authored-by: Sergey Polzunov <traut@users.noreply.github.com>

* [FR] Bump changed-files Version to Patched Version (elastic#4542)

* Bump changed-files Version to Patched Version

* patch bump

* reenable workflow

* Use full length commit hash

* Bump 44 to 46

---------

Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>

* [New Rule] Adding Coverage for DynamoDB Exfiltration Behaviors (elastic#4535)

* new rules for AWS DynamoDB data exfiltration

* bumping patch version

* adjusting investigation guide

* updating patch version

* updating patch version

* updating patch version

---------

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>

* fix: removing outdated code in Kibana client auth (elastic#4495)

* Simplify kibana session management

* Drop removed options from `kibana_args` set

* Style fix

* Patch version bump

* Bumping kibana lib version

* Relax CLI requirement, making `api_key` optional, to allow `help` to run

* Create new detection rule set documentation to be included in the new docs. (elastic#4508)

* move docs folder to docs-dev

* Add new docs folder

* update docset.yml to reflect latest usage

* Add rules_building_block folder

* revert changes to docs-dev/experimental-machine-learning/url-spoof.md

* bump patch versions

* revert bump

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>

* fixing double header in investigation notes (elastic#4490)

* [Bug] Update Custom Rules Markdown Location (elastic#4565)

* Update to custom-rules markdown location

* bump version

* Update link reference

* Prep main for 9.1 (elastic#4555)

* Prep for Release 9.1

* Update Patch Version

* Update Patch version

* Update Patch version

* [Rule Tuning] Added OWA (outlook for web) new AppID (elastic#4568)

* Added OWA (outlook for web) new AppID

**Title:** Add new Outlook for Web AppID to abnormal Microsoft 365 ClientAppID rule

**Description:**

This pull request updates the `initial_access_microsoft_365_abnormal_clientappid` rule to include the newly introduced Outlook for Web AppID:
- **New AppID**: `9199bf20-a13f-4107-85dc-02114787ef48`

### Context

Outlook for Web (OWA) is migrating to a new authentication platform using MSAL and a Single Page Application (SPA) auth model. As part of this backend change, Microsoft is replacing the existing OWA AppID with a new one. This change is being rolled out during the first half of calendar year 2024, with full deployment expected by Q4 2024.
- **Old OWA AppID**: `00000002-0000-0ff1-ce00-000000000000`
- **New OWA AppID**: `9199bf20-a13f-4107-85dc-02114787ef48`
    

Although no action is required for tenant administrators, this new AppID may show up in logs and should be accounted for in detections relying on known legitimate ClientAppIDs.

### Why this change?

The rule `initial_access_microsoft_365_abnormal_clientappid` flags potentially suspicious or unauthorized client applications accessing Microsoft 365 services. To prevent false positives caused by this official change from Microsoft, this PR adds the new OWA AppID to the allowlist.

### References
- Microsoft 365 Message Center notice (ref: MC715025)
- [MSAL documentation](https://learn.microsoft.com/en-us/azure/active-directory/develop/msal-overview)

* Update initial_access_microsoft_365_abnormal_clientappid.toml

Updated updated_date

* Update Max signals value to supported limits (elastic#4556)

* Update ATT&CK coverage URL(s) in docs-dev/ATT&CK-coverage.md (elastic#4571)

* deprecating 'Azure Virtual Network Device Modified or Deleted' (elastic#4559)

* tuning 'Azure Conditional Access Policy Modified' (elastic#4558)

* [Rule Tuning] Tuning Illicit Grant Consent Detections in Azure and M365 (elastic#4557)

* tuning Azure rule for illicit grant activity; creating new rule for M365

* Update rules/integrations/o365/initial_access_microsoft_365_illicit_consent_grant_via_registered_application.toml

* Update rules/integrations/azure/initial_access_entra_illicit_consent_grant_via_registered_application.toml

* adjusted tags

* Update rules/integrations/azure/initial_access_entra_illicit_consent_grant_via_registered_application.toml

* Update defense_evasion_microsoft_defender_tampering.toml (elastic#4573)

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* [Rule Tuning] O365 Exchange Suspicious Mailbox Right Delegation (elastic#4589)

* Update docset.yml (elastic#4590)

Remove diagnostic hint

* [New] Unusual Network Connection to Suspicious Web Service (elastic#4569)

* [New] Unusual Network Connection to Suspicious Web Service

* Update rule threat order

---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* [New] Unusual Network Connection to Suspicious Top Level Domain (elastic#4563)

* [Rule Tuning] Add Host Metadata to ES|QL Aggregation Rules (elastic#4592)

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Add investigation guides (elastic#4600)

* [Rule Tuning] Suspicious Execution via Scheduled Task (elastic#4599)

* Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (elastic#4601)

* [FN Tuning] Shared Object Created or Changed by Previously Unknown Pr… (elastic#4529)

* [FN Tuning] Shared Object Created or Changed by Previously Unknown Process

* Update process exclusions in TOML file

---------

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

* [D4C Conversion] Converting Compatible D4C Rules to DR (elastic#4532)

* [D4C Conversion] Converting Compatible D4C Rules to DR

* added host.os.type

* Rename

* Update rules/linux/execution_container_management_binary_launched_inside_container.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/linux/privilege_escalation_debugfs_launched_inside_container.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/linux/privilege_escalation_debugfs_launched_inside_container.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/linux/privilege_escalation_mount_launched_inside_container.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/linux/privilege_escalation_mount_launched_inside_container.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

---------

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Remove Task List reference (elastic#4605)

* [FR] Update Detection Rules MITRE Workflow to SHA Pin  (elastic#4581)

* Update to pinned hash

* version bump

* [FR] Add Kibana Action Connector Error to Exception List Workaround (elastic#4583)

* Add error catch for workaround

* Switch to set for efficiency

* Patch version bump

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* [Rule Tuning] SSH Authorized Keys File Deletion (elastic#4591)

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* [Rule Tuning] Suspicious WMI Event Subscription Created (elastic#4618)

* [Rule Tuning] Suspicious Execution via Scheduled Task

* [Rule Tuning] Suspicious WMI Event Subscription Created

* [Rule Tuning] Adjusting `Microsoft Entra ID Rare Authentication Requirement for Principal User` (elastic#4562)

* tuning 'Microsoft Entra ID Rare Authentication Requirement for Principal User'

* updated MITRE ATT&CK mappings

* updated index target

* updated patch version

* updating patch version

* bumping patch version

* updating patch version

* [Rule Tuning] Tuning `Azure Service Principal Credentials Added` (elastic#4570)

* tuning 'Azure Service Principal Credentials Added'

* updated patch version

* added investigation guide

* updating patch version

* updating patch version

* [FR] Add Support for Local Dates Flag (elastic#4582)

* Add support for local dates flag

* Use two variables

* Add support for import-rules-to-repo

* Revert arg formatting

* Update comment

* Pass Rule Path as Path Object

* Update to rule loader function

* Streamline metadata function

* Also support dictionaries

* Bump patch version

* Reduce complexity

* Add if path exists check

* Fix version bump

* Feature exclude tactic name (elastic#4593)

* Added new cli flag to exclude tactic name in rule file name

* added a shortcut for the flag and adjusted CLI readme

* Add no tactic flag also to import to prevent warnings

* Added info about unit test

* version bump

* Added no_tactic_filename as config option + fixed linting

* pyproject version bump

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: Kirti Sodhi <109447885+sodhikirti07@users.noreply.github.com>
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
Co-authored-by: Martijn Laarman <Mpdreamz@gmail.com>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Jan Calanog <nejcalanog@gmail.com>
Co-authored-by: Sergey Polzunov <traut@users.noreply.github.com>
Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
Co-authored-by: M. Visser <BugOrFeature@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>
Co-authored-by: Frederik Berg <83548283+frederikb96@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shashank-elastic shashank-elastic

@Mikaayenson Mikaayenson

@eric-forte-elastic eric-forte-elastic

@terrancedejesus terrancedejesus

Assignees
No one assigned
Labels
backport: auto bug Something isn't working major python Internal python for the repository
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@traut @eric-forte-elastic @Mikaayenson @shashank-elastic