Disable normalization validating fields in transforms if synthetics is enabled #2011

mrodm · 2024-08-09T15:23:54Z

Relates #1971

Fields in transforms were always validated using normalization.

This PR allows to disable normalization when validating those fields (from transforms) when synthetic is enabled in the data stream created in testing.

Example of errors found in ti_anomali package:

test case failed: one or more errors found in documents stored in logs-ti_anomali.threatstream-77107 data stream: [0] field "event.category" is not normalized as expected: expected array, found "threat" (string)
[1] field "event.type" is not normalized as expected: expected array, found "indicator" (string)
[2] field "labels.is_ioc_transform_source" is undefined

Validation for event.type and event.category fields is happening when checking transforms. If checkTransforms method is commented out, validation for those fields run successfully.

As a note, labels.is_ioc_transform_source fields still fails even with this fix. Probably, some missing mapping/field definition? why is it not failing with logsDB disabled ?

Author's checklist

Remove setting logsDB as the default mode.
Support failure store when synthetics (logsDB) is enabled.
Update ti_anomali with contents from integrations.
Checked that ti_anomali_logsdb package has the validation errors if normalization is not disabled.
- Added missing labels.is_ioc_transform_source field definition in transform.

…orms

mrodm · 2024-08-09T15:26:36Z

internal/testrunner/runners/system/tester.go

@@ -1462,7 +1462,7 @@ func (r *tester) validateTestScenario(ctx context.Context, result *testrunner.Re
 	}

 	// Check transforms if present
-	if err := r.checkTransforms(ctx, config, r.pkgManifest, scenario.kibanaDataStream, scenario.dataStream); err != nil {
+	if err := r.checkTransforms(ctx, config, r.pkgManifest, scenario.kibanaDataStream, scenario.dataStream, scenario.syntheticEnabled); err != nil {


Use the same value as in the main data stream.
Would that assumption be right?

Not sure what Fleet does with transform indexes. Could we add a test case that would fail without this change?

I'll check to run a test package for that use case (with logsdb enabled).

Added a new test package ti_anomali_logsdb copied from ti_anomali. This allows to test the package with both logsdb enabled and disabled.

mrodm · 2024-08-09T15:43:11Z

test integrations

elastic-vault-github-plugin-prod · 2024-08-09T15:48:42Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10752

mrodm · 2024-08-12T08:59:55Z

With the changes of synthetics while transforms are checked, those values are not valid:

mrodm · 2024-08-12T09:37:07Z

Failure store is failing when synthetics is enabled when unmarshalling the response (error in CI link):

Error: error running package system tests: could not complete test run: failed to check failure store: failed to decode search response: json: cannot unmarshal string into Go struct field .hits.hits._source.error.pipeline_trace of type []string

mrodm · 2024-08-12T10:25:15Z

test integrations

elastic-vault-github-plugin-prod · 2024-08-12T10:30:05Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10752

mrodm · 2024-08-12T13:48:31Z

internal/testrunner/runners/system/tester.go

+	case string:
+		*p = append(*p, v)
+	case []any:
+		// asume it is going to be an array of strings
+		for _, value := range v {
+			*p = append(*p, fmt.Sprint(value))
+		}


Support string and slice of strings.

Examples of the response:

logsdb disabled in the stack:
{ "type": "illegal_argument_exception", "message": "unable to convert [eight] to integer", "stack_trace": "...", "pipeline_trace": [ "logs-failure_store.test-0.0.1" ], "pipeline": "logs-failure_store.test-0.0.1", "processor_type": "convert" }

logsdb enabled in the stack:
{ "message": "unable to convert [eight] to integer", "pipeline": "logs-failure_store.test-0.0.1", "pipeline_trace": "logs-failure_store.test-0.0.1", "processor_type": "convert", "stack_trace": "...", "type": "illegal_argument_exception" }

mrodm · 2024-08-12T13:54:44Z

Integrations build running with Elastic stack 8.15.0-SNAPSHOT and logsdb enabled: https://buildkite.com/elastic/integrations/builds/14572

In that build, ti_anomali just fails with the labels.is_ioc_transform_source. The other two fields (event.category and event.type) are valid now.

This reverts commit ab88d3d.

mrodm · 2024-08-12T13:57:04Z

test integrations

elastic-vault-github-plugin-prod · 2024-08-12T14:02:21Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10752

jsoriano

Changes make sense to me, but let's try to find a test case that can help to confirm if transforms have the same index mode as the data streams of the package.

jsoriano · 2024-08-13T10:36:25Z

internal/testrunner/runners/system/tester.go

@@ -1462,7 +1462,7 @@ func (r *tester) validateTestScenario(ctx context.Context, result *testrunner.Re
 	}

 	// Check transforms if present
-	if err := r.checkTransforms(ctx, config, r.pkgManifest, scenario.kibanaDataStream, scenario.dataStream); err != nil {
+	if err := r.checkTransforms(ctx, config, r.pkgManifest, scenario.kibanaDataStream, scenario.dataStream, scenario.syntheticEnabled); err != nil {


Not sure what Fleet does with transform indexes. Could we add a test case that would fail without this change?

…est package

…mplate test package" This reverts commit 708f4c8.

elasticmachine · 2024-08-13T18:40:01Z

💚 Build Succeeded

Buildkite Build
Commit: 6ed4ed8

History

💔 Build #3724 failed bb32496
💔 Build #3722 failed 21e4f27
💚 Build #3712 succeeded 0c8abda
💚 Build #3710 succeeded 1003403
💔 Build #3705 failed ab88d3d

cc @mrodm

mrodm · 2024-08-14T08:17:11Z

scripts/stack_parameters.sh

Added a new file to be able "source" it in the script in charge of testing false positives (test-check-false-positives.sh) and the generic one (test-check-packages.sh).

mrodm · 2024-08-14T08:17:42Z

scripts/test-check-false-positives.sh

+stack_args=$(stack_version_args) # --version <version>

 # Update the stack
 elastic-package stack update -v ${stack_args}


elastic-package stack update command does not allow the -U parameter.

mrodm · 2024-08-14T08:18:17Z

test/packages/parallel/ti_anomali/_dev/build/docs/README.md

Updated test package with the latest contents from integrations repository.

mrodm · 2024-08-14T08:21:25Z

test/packages/parallel/ti_anomali_logsdb/_dev/build/docs/README.md

In order to be able to test ti_anomali with both logsdb enabled and disabled, a new package ti_anomali_logsdb has been added (copied from ti_anomali).

This new package is tested using a Elastic stack with the option -U stack.logsdb_enabled=true, thanks to the file test/packages/parallel/ti_anomali_logsdb.stack_provider_settings

mrodm · 2024-08-14T08:59:02Z

test integrations

elastic-vault-github-plugin-prod · 2024-08-14T09:04:47Z

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10752

mrodm added 2 commits August 9, 2024 17:07

Disable normalization if synthetics is enabled when validating transf…

84fc9ac

…orms

Enable logsdb by default - to be reverted

ab88d3d

mrodm self-assigned this Aug 9, 2024

mrodm commented Aug 9, 2024

View reviewed changes

mrodm changed the title ~~Disable normalization validating transforms synthetics~~ Disable normalization validating fields in transforms if synthetics is enabled Aug 9, 2024

mrodm mentioned this pull request Aug 9, 2024

Incorrect validation of documents with arrays of strings in multi-fields #1971

Closed

elastic-vault-github-plugin-prod bot mentioned this pull request Aug 9, 2024

Test elastic-package#2011 - DO NOT MERGE elastic/integrations#10752

Closed

Allow to read pipeline trace value as string or slice of strings

1003403

mrodm commented Aug 12, 2024

View reviewed changes

Revert "Enable logsdb by default - to be reverted"

0c8abda

This reverts commit ab88d3d.

mrodm marked this pull request as ready for review August 13, 2024 10:28

mrodm requested a review from a team August 13, 2024 10:29

jsoriano reviewed Aug 13, 2024

View reviewed changes

mrodm added 6 commits August 13, 2024 13:36

Add error and expected errors in rally_benchmark_malformed_template t…

708f4c8

…est package

Add test package to catch errors in transforms

5e96fbc

Revert "Add error and expected errors in rally_benchmark_malformed_te…

21e4f27

…mplate test package" This reverts commit 708f4c8.

Update ti_anomali and ti_anomali_logsdb packages

bb32496

Allow to be undefined PACKAGE_UNDER_TEST var

fc042f2

Just define provider args in elastic-package stack up command

6ed4ed8

mrodm commented Aug 14, 2024

View reviewed changes

mrodm requested a review from jsoriano August 14, 2024 10:49

jsoriano approved these changes Aug 14, 2024

View reviewed changes

mrodm merged commit 3969afa into elastic:main Aug 14, 2024
3 checks passed

mrodm deleted the disable_normalization_validating_transforms_synthetics branch August 14, 2024 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable normalization validating fields in transforms if synthetics is enabled #2011

Disable normalization validating fields in transforms if synthetics is enabled #2011

mrodm commented Aug 9, 2024 •

edited

Loading

mrodm Aug 9, 2024 •

edited

Loading

jsoriano Aug 13, 2024

mrodm Aug 13, 2024

mrodm Aug 14, 2024

mrodm commented Aug 9, 2024

elastic-vault-github-plugin-prod bot commented Aug 9, 2024

mrodm commented Aug 12, 2024

mrodm commented Aug 12, 2024

mrodm commented Aug 12, 2024

elastic-vault-github-plugin-prod bot commented Aug 12, 2024

mrodm Aug 12, 2024 •

edited

Loading

mrodm commented Aug 12, 2024

mrodm commented Aug 12, 2024

elastic-vault-github-plugin-prod bot commented Aug 12, 2024

jsoriano left a comment

jsoriano Aug 13, 2024

elasticmachine commented Aug 13, 2024

mrodm Aug 14, 2024

mrodm Aug 14, 2024

mrodm Aug 14, 2024

mrodm Aug 14, 2024

mrodm commented Aug 14, 2024

elastic-vault-github-plugin-prod bot commented Aug 14, 2024

Disable normalization validating fields in transforms if synthetics is enabled #2011

Disable normalization validating fields in transforms if synthetics is enabled #2011

Conversation

mrodm commented Aug 9, 2024 • edited Loading

Author's checklist

mrodm Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm commented Aug 9, 2024

elastic-vault-github-plugin-prod bot commented Aug 9, 2024

mrodm commented Aug 12, 2024

mrodm commented Aug 12, 2024

mrodm commented Aug 12, 2024

elastic-vault-github-plugin-prod bot commented Aug 12, 2024

mrodm Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

mrodm commented Aug 12, 2024

mrodm commented Aug 12, 2024

elastic-vault-github-plugin-prod bot commented Aug 12, 2024

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 13, 2024

💚 Build Succeeded

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrodm commented Aug 14, 2024

elastic-vault-github-plugin-prod bot commented Aug 14, 2024

mrodm commented Aug 9, 2024 •

edited

Loading

mrodm Aug 9, 2024 •

edited

Loading

mrodm Aug 12, 2024 •

edited

Loading