Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG][Serverless][8.15 & 8.16] Refresh the Optimizing anomaly results topic #5739

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified docs/advanced-entity-analytics/images/filter-add-item.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
86 changes: 40 additions & 46 deletions docs/advanced-entity-analytics/tune-anomaly-results.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,40 +16,39 @@ you can filter out the unwanted results.
For example, to filter out results from a housekeeping process, named
`maintenanceservice.exe`, that only executes occasionally you need to:

. <<create-fiter-list>>
. <<create-filter-list>>
. <<add-job-filter>>
. <<clone-job, Clone and rerun the job>> (optional)

[float]
[[create-fiter-list]]
[[create-filter-list]]
//Make sure that fixing this typo doesn't affect any other references in the Security docset and elsewhere.
=== Create a filter list

. Go to *Machine Learning* -> *Anomaly Detection* -> *Settings*.
. Click *Filter Lists* and then *New*.
+
The *Create new filter list* pane is displayed.
. Enter a filter list ID.
. Enter a description for the filter list (optional).
. Click *Add item*.
. In the *Items* textbox, enter the name of the process for which you want to
filter out anomaly results (`maintenanceservice.exe` in our example).
. To begin creating a new filter, go to **Kibana**, then **Machine Learning** -> **Anomaly Detection** -> **Settings**.
. In the **Filter Lists** section, click **Create**.
. On the Create new filter list page, complete the following:
.. Enter an ID to name the filter list.
.. (Optional) Provide a description for the filter list.
.. Specify the processes that you want to filter out from anomaly results. To do this, click *Add item*, then enter processes names in the text box. In the example below, the `maintenanceservice.exe` process is being added to a filter list that specifies processes to filter out from anomaly results.

+
[role="screenshot"]
image::filter-add-item.png[]
. Click *Add* and then *Save*.
+
The new filter appears in the Filter List and can be added to relevant jobs.
.. Click *Add* and then *Save*.

The new filter appears on the Filter Lists page, where you can add it to relevant jobs.

[float]
[[add-job-filter]]
=== Add the filter to the relevant job

. Go to *Machine Learning* -> *Anomaly Detection* -> *Anomaly Explorer*.
. Navigate to the job results for which the filter is required. If the job results
are not listed, click *Edit job selection* and select the relevant job.
. In the *actions* column, click the gear icon and then select _Configure rules_.
. In Kibana, go to **Machine Learning** -> **Anomaly Detection** -> **Anomaly Explorer**.
. In the Job selection flyout, select the jobs for which you want to add a filter to. If the jobs don't have results, click **Edit job selection* to select other jobs.
. Go Anomalies section of the Anomaly Explorer page and and navigate to the job results for which the filter is required.
. In the **Actions** column, click the gear icon, then select **Configure rules**.
+
The *Create Rule* window is displayed.
The Create Rule window displays.
+
[role="screenshot"]
image::rule-scope.png[]
Expand All @@ -58,13 +57,13 @@ image::rule-scope.png[]
.. The _WHEN_ statement for the relevant detector (`process.name` in our
example).
.. The _IS IN_ statement.
.. The filter you created as part of the <<create-fiter-list>> procedure.
.. The filter you created as part of the <<create-filter-list>> procedure.
+
TIP: For more information, see
TIP: To learn more about creating filters that change the behavior of anomaly detectors, refer to
{ml-docs}/ml-configuring-detector-custom-rules.html[Customizing detectors with custom rules].

. Click *Save*.

. Click *Save* to save the filter to the job results.
+
NOTE: Changes to rules only affect new results. All anomalies found by the job
before the filter was added are still displayed.

Expand All @@ -78,33 +77,27 @@ must clone and run the cloned job.
IMPORTANT: Running the cloned job can take some time. Only run the job after you
have completed all job rule changes.

. Go to *Machine Learning* -> *Anomaly Detection* -> *Job Management*.
. Navigate to the job for which you configured the rule.
. Optionally, expand the job row and click *JSON* to verify the configured filter
appears under `custom rules` in the JSON code.
. In the *actions* column, click the more (three dots) icon and select _Clone job_.
+
The *Configure datafeed* page is displayed.
. Click *Data Preview* and check the data is displayed without errors.
. Click *Next* until the *Job details* page is displayed.
. Enter a Job ID for the cloned job that indicates it is an iteration of the
original one. For example, append a number or a username to the original job
name, such as `windows-rare-network-process-2`.
. In Kibana, go to *Machine Learning* -> *Anomaly Detection* -> *Jobs*.
. Navigate to the job for which you configured the rule. Optionally, expand the job row and go to the *JSON* tab to verify the configured filter appears under `custom rules` in the JSON code.
. In the *Actions* column, click the **All actions** menu (**...**), then select **Clone job**. The **Create job: Advanced** page is displays.
. Click **Next** until you get to the Job details page.
. Enter a job ID for the cloned job. We recommend creating a name that shows the new job is an iteration of the original one. For example, append a number or a username to the original job name, such as `windows-rare-network-process-2`.
+
[role="screenshot"]
image::cloned-job-details.png[]
. Click *Next* and check the job validates without errors. You can ignore
warnings about multiple influencers.
. Click *Next* and then *Create job*.

. Click **Next** and confirm that the job doesn't return errors. You can ignore warnings about multiple influencers.
. Click **Next**, then **Create job**.
+
The *Start <job name>* window is displayed.
The *Start <job name>* window displays.
+
[role="screenshot"]
image::start-job-window.png[]
. Select the point of time from which the job will analyze anomalies.
. Click *Start*.

. Specify when the job begins to analyze anomalies.
. Click **Start**.
+
After a while, results will start to appear on the *Anomaly Explorer* page.
Results will eventually appear on the Anomaly Explorer page.

[float]
[[define-rule-threshold]]
Expand All @@ -121,7 +114,7 @@ Depending on your anomaly detection results, you may want to set a
minimum event count threshold for the `packetbeat_dns_tunneling` job:


. Go to *Machine Learning* -> *Anomaly Detection* -> *Anomaly Explorer*.
. In Kibana, go to **Machine Learning** -> **Anomaly Detection** -> **Anomaly Explorer**.
. Navigate to the job results for the `packetbeat_dns_tunneling` job. If the
job results are not listed, click *Edit job selection* and select
`packetbeat_dns_tunneling`.
Expand All @@ -138,6 +131,7 @@ image::ml-rule-threshold.png[]
_WHEN actual IS GREATER THAN <X>_
+
Where `<X>` is the threshold above which anomalies are detected.
. Click *Save*.
. To apply the new threshold, rerun the job (*Job Management* -> *Actions* ->
*Start datafeed*).
. Click **Save**.
. To apply the new threshold, rerun the job:
.. Go to **Anomaly Detection** -> **Jobs**, and find the `packetbeat_dns_tunneling`.
.. In the *Actions* column, click the **All actions** menu (**...**), then select **Start datafeed**.