Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for multiple regex in a search #3413

Closed
wants to merge 5 commits into from

Conversation

stevengoossensB
Copy link

@stevengoossensB stevengoossensB commented Jan 31, 2024

Summary

Allow for KQL searches such as field: * my value *

Add search term between wildcards e.g *my term*
@Mikaayenson
Copy link
Contributor

👋 Thanks for submitting the PR @stevengoossensB. Can you add test cases to this PR?

Updated test case for a.keyword:*hello world*
@stevengoossensB
Copy link
Author

I've added the test cases for this change

@Mikaayenson
Copy link
Contributor

Thanks @stevengoossensB. It looks like your PR is failing the CI:

=================================== FAILURES ===================================
___________________________ ParserTests.test_keyword ___________________________

self = <tests.kuery.test_parser.ParserTests testMethod=test_keyword>

    def test_keyword(self):
        schema = {
            "a.text": "text",
            "a.keyword": "keyword",
            "b": "long",
        }
    
        self.validate('a.text:hello', FieldComparison(Field("a.text"), String("hello")), schema=schema)
        self.validate('a.keyword:hello', FieldComparison(Field("a.keyword"), String("hello")), schema=schema)
    
        self.validate('a.text:"hello"', FieldComparison(Field("a.text"), String("hello")), schema=schema)
        self.validate('a.keyword:"hello"', FieldComparison(Field("a.keyword"), String("hello")), schema=schema)
    
        self.validate('a.text:1', FieldComparison(Field("a.text"), String("1")), schema=schema)
        self.validate('a.keyword:1', FieldComparison(Field("a.keyword"), String("1")), schema=schema)
    
        self.validate('a.text:"1"', FieldComparison(Field("a.text"), String("1")), schema=schema)
        self.validate('a.keyword:"1"', FieldComparison(Field("a.keyword"), String("1")), schema=schema)
    
>       self.validate('a.text:*hello world*', FieldComparison(Field("a.text"), String("1")), schema=schema)

tests/kuery/test_parser.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/kuery/test_parser.py:22: in validate
    self.assertEqual(kql.parse(source, *args, **kwargs), tree)
kql/__init__.py:53: in parse
    converted = KqlParser(text, schema=schema).visit(lark_parsed)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/lark/visitors.py:381: in visit
    return f(tree)
kql/parser.py:313: in field_value_expression
    return FieldComparison(field, self.visit(expr))
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/lark/visitors.py:381: in visit
    return f(tree)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kql.parser.KqlParser object at 0x7f3559458250>
tree = Tree('value', [Token('UNQUOTED_LITERAL', '*hello world*')])

    def value(self, tree):
        if self.scoped_field is None:
            raise self.error(tree, "Value not tied to field")
    
        field_name = self.scoped_field.name
        token = tree.children[0]
        value = self.unescape_literal(token)
    
        if token.type == "UNQUOTED_LITERAL" and "*" in token.value:
            field_type = self.get_field_type(field_name)
            if len(value.replace("*", "")) == 0:
                return Exists()
    
            if field_type is not None and field_type not in ("keyword", "wildcard"):
>               raise self.error(tree, "Unable to perform wildcard on field {field} of {type}",
                                 field=field_name, type=field_type)
E               kql.errors.KqlParseError: Error at line:1,column:8
E               Unable to perform wildcard on field a.text of text
E               a.text:*hello world*
E                      ^^^^^^^^^^^^^

kql/parser.py:354: KqlParseError
________________ TestValidRules.test_all_rule_queries_optimized ________________
tests/base.py:71: in setUp
    self.fail(f'Rule loader failure: \n{RULE_LOADER_FAIL_MSG}')
E   AssertionError: Rule loader failure: 
E   Error at line:2,column:99
E   Invalid syntax
E   event.category:file and event.type:change and
E     (file.name:pam_*.so or file.path:(/etc/pam.d/* or /private/etc/pam.d/* or /usr/lib64/security/*)) and
E                                                                                                     ^
---------------------------- Captured stdout setup -----------------------------
Error loading rule in /home/runner/work/detection-rules/detection-rules/rules/cross-platform/persistence_credential_access_modify_auth_module_or_config.toml
=============================== warnings summary ===============================
../../../../../opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/config/__init__.py:1223
  /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/config/__init__.py:1223: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: typeguard
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/kuery/test_parser.py::ParserTests::test_keyword - kql.errors.KqlParseError: Error at line:1,column:8
Unable to perform wildcard on field a.text of text
a.text:*hello world*
       ^^^^^^^^^^^^^
FAILED tests/test_all_rules.py::TestValidRules::test_all_rule_queries_optimized - AssertionError: Rule loader failure: 
Error at line:2,column:99
Invalid syntax
event.category:file and event.type:change and
  (file.name:pam_*.so or file.path:(/etc/pam.d/* or /private/etc/pam.d/* or /usr/lib64/security/*)) and
                                                                                                  ^
============= 2 failed, 83 passed, 56 skipped, 1 warning in 12.82s =============
Error: Process completed with exit code 1.

stevengoossensB and others added 2 commits February 5, 2024 16:05
Remove text sample as no wildcards are allowed there
Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there are still some failing tests based on this change.

Failing CI

Run python -m detection_rules test

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-8.0.0, pluggy-1.4.0 -- /opt/hostedtoolcache/Python/3.8.18/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/detection-rules/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collecting ... collected 141 items

tests/kuery/test_dsl.py::TestKQLtoDSL::test_and_query PASSED             [  0%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_exists PASSED          [  1%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_inequality PASSED      [  2%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_match PASSED           [  2%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_not_query PASSED             [  3%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_optimizations PASSED         [  4%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_or_query PASSED              [  4%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_and_query PASSED          [  5%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_boolean_precedence PASSED [  6%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_field_equals PASSED       [  7%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_field_inequality PASSED   [  7%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_ip_checks PASSED          [  8%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_list_of_values PASSED     [  9%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_not_query PASSED          [  9%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_or_query PASSED           [ 10%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_wildcard_field PASSED     [ 11%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_and_expr PASSED      [ 12%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_and_values PASSED    [ 12%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_cidr_match PASSED    [ 13%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_field_exists PASSED  [ 14%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_flattening PASSED    [ 14%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_list_value PASSED    [ 15%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_not_value PASSED     [ 16%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_or_expr PASSED       [ 17%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_or_values PASSED     [ 17%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_quoted_wildcard PASSED [ 18%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_range PASSED         [ 19%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_single_value PASSED  [ 19%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_wildcard PASSED      [ 20%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_and_query PASSED          [ 21%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_boolean_precedence PASSED [ 21%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_field_equals PASSED       [ 22%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_field_inequality PASSED   [ 23%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_list_of_values PASSED     [ 24%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_lone_value PASSED         [ 24%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_nested_query PASSED       [ 25%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_not_query PASSED          [ 26%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_or_query PASSED           [ 26%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_schema PASSED             [ 27%]
tests/kuery/test_lint.py::LintTests::test_and_not PASSED                 [ 28%]
tests/kuery/test_lint.py::LintTests::test_compound PASSED                [ 29%]
tests/kuery/test_lint.py::LintTests::test_double_negate PASSED           [ 29%]
tests/kuery/test_lint.py::LintTests::test_extract_not PASSED             [ 30%]
tests/kuery/test_lint.py::LintTests::test_ip PASSED                      [ 31%]
tests/kuery/test_lint.py::LintTests::test_lint_field PASSED              [ 31%]
tests/kuery/test_lint.py::LintTests::test_lint_precedence PASSED         [ 32%]
tests/kuery/test_lint.py::LintTests::test_merge_fields PASSED            [ 33%]
tests/kuery/test_lint.py::LintTests::test_mixed_demorgans PASSED         [ 34%]
tests/kuery/test_lint.py::LintTests::test_not_demorgans PASSED           [ 34%]
tests/kuery/test_lint.py::LintTests::test_not_or PASSED                  [ 35%]
tests/kuery/test_lint.py::LintTests::test_upper_tokens PASSED            [ 36%]
tests/kuery/test_parser.py::ParserTests::test_conversion PASSED          [ 36%]
tests/kuery/test_parser.py::ParserTests::test_date PASSED                [ 37%]
tests/kuery/test_parser.py::ParserTests::test_keyword FAILED             [ 38%]
tests/kuery/test_parser.py::ParserTests::test_list_equals PASSED         [ 39%]
tests/kuery/test_parser.py::ParserTests::test_multiple_types_fail PASSED [ 39%]
tests/kuery/test_parser.py::ParserTests::test_multiple_types_success PASSED [ 40%]
tests/kuery/test_parser.py::ParserTests::test_number_exists PASSED       [ 41%]
tests/kuery/test_parser.py::ParserTests::test_number_wildcard_fail PASSED [ 41%]
tests/kuery/test_parser.py::ParserTests::test_type_family_fail PASSED    [ 42%]
tests/kuery/test_parser.py::ParserTests::test_type_family_success PASSED [ 43%]
tests/test_all_rules.py::TestValidRules::test_all_rule_queries_optimized FAILED [ 43%]
tests/test_all_rules.py::TestValidRules::test_bbr_validation SKIPPED     [ 44%]
tests/test_all_rules.py::TestValidRules::test_duplicate_file_names SKIPPED [ 45%]
tests/test_all_rules.py::TestValidRules::test_file_names SKIPPED (Ru...) [ 46%]
tests/test_all_rules.py::TestValidRules::test_production_rules_have_rta SKIPPED [ 46%]
tests/test_all_rules.py::TestValidRules::test_rule_type_changes SKIPPED  [ 47%]
tests/test_all_rules.py::TestValidRules::test_schema_and_dupes SKIPPED   [ 48%]
tests/test_all_rules.py::TestThreatMappings::test_duplicated_tactics SKIPPED [ 48%]
tests/test_all_rules.py::TestThreatMappings::test_tactic_to_technique_correlations SKIPPED [ 49%]
tests/test_all_rules.py::TestThreatMappings::test_technique_deprecations SKIPPED [ 50%]
tests/test_all_rules.py::TestRuleTags::test_casing_and_spacing SKIPPED   [ 51%]
tests/test_all_rules.py::TestRuleTags::test_investigation_guide_tag SKIPPED [ 51%]
tests/test_all_rules.py::TestRuleTags::test_ml_rule_type_tags SKIPPED    [ 52%]
tests/test_all_rules.py::TestRuleTags::test_no_duplicate_tags SKIPPED    [ 53%]
tests/test_all_rules.py::TestRuleTags::test_os_tags SKIPPED (Rule lo...) [ 53%]
tests/test_all_rules.py::TestRuleTags::test_primary_tactic_as_tag SKIPPED [ 54%]
tests/test_all_rules.py::TestRuleTags::test_required_tags SKIPPED (R...) [ 55%]
tests/test_all_rules.py::TestRuleTags::test_tag_prefix SKIPPED (Rule...) [ 56%]
tests/test_all_rules.py::TestRuleTimelines::test_timeline_has_title SKIPPED [ 56%]
tests/test_all_rules.py::TestRuleFiles::test_bbr_in_correct_dir SKIPPED  [ 57%]
tests/test_all_rules.py::TestRuleFiles::test_non_bbr_in_correct_dir SKIPPED [ 58%]
tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic SKIPPED [ 58%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules SKIPPED [ 59%]
tests/test_all_rules.py::TestRuleMetadata::test_event_dataset SKIPPED    [ 60%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag SKIPPED  [ 60%]
tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries SKIPPED  [ 61%]
tests/test_all_rules.py::TestRuleMetadata::test_updated_date_newer_than_creation SKIPPED [ 62%]
tests/test_all_rules.py::TestIntegrationRules::test_all_min_stack_rules_have_comment SKIPPED [ 63%]
tests/test_all_rules.py::TestIntegrationRules::test_integration_guide SKIPPED [ 63%]
tests/test_all_rules.py::TestIntegrationRules::test_ml_integration_jobs_exist SKIPPED [ 64%]
tests/test_all_rules.py::TestIntegrationRules::test_rule_demotions SKIPPED [ 65%]
tests/test_all_rules.py::TestRuleTiming::test_eql_interval_to_maxspan SKIPPED [ 65%]
tests/test_all_rules.py::TestRuleTiming::test_eql_lookback SKIPPED (...) [ 66%]
tests/test_all_rules.py::TestRuleTiming::test_event_override SKIPPED     [ 67%]
tests/test_all_rules.py::TestRuleTiming::test_required_lookback SKIPPED  [ 68%]
tests/test_all_rules.py::TestLicense::test_elastic_license_only_v2 SKIPPED [ 68%]
tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields SKIPPED [ 69%]
tests/test_all_rules.py::TestBuildTimeFields::test_build_fields_min_stack SKIPPED [ 70%]
tests/test_all_rules.py::TestRiskScoreMismatch::test_rule_risk_score_severity_mismatch SKIPPED [ 70%]
tests/test_all_rules.py::TestNoteMarkdownPlugins::test_if_plugins_explicitly_defined SKIPPED [ 71%]
tests/test_all_rules.py::TestNoteMarkdownPlugins::test_note_has_osquery_warning SKIPPED [ 72%]
tests/test_all_rules.py::TestNoteMarkdownPlugins::test_plugin_placeholders_match_entries SKIPPED [ 73%]
tests/test_all_rules.py::TestAlertSuppression::test_group_field_in_schemas SKIPPED [ 73%]
tests/test_all_rules.py::TestAlertSuppression::test_group_length SKIPPED [ 74%]
tests/test_gh_workflows.py::TestWorkflows::test_matrix_to_lock_version_defaults PASSED [ 75%]
tests/test_mappings.py::TestMappings::test_false_positives SKIPPED (...) [ 75%]
tests/test_mappings.py::TestMappings::test_true_positives SKIPPED (R...) [ 76%]
tests/test_mappings.py::TestRTAs::test_rtas_with_triggered_rules_have_uuid PASSED [ 77%]
tests/test_packages.py::TestPackages::test_package_loader_default_configs SKIPPED [ 78%]
tests/test_packages.py::TestPackages::test_package_loader_production_config SKIPPED [ 78%]
tests/test_packages.py::TestPackages::test_package_summary SKIPPED (...) [ 79%]
tests/test_packages.py::TestPackages::test_rule_versioning SKIPPED (...) [ 80%]
tests/test_packages.py::TestRegistryPackage::test_registry_package_config PASSED [ 80%]
tests/test_schemas.py::TestSchemas::test_eql_validation PASSED           [ 81%]
tests/test_schemas.py::TestSchemas::test_query_downgrade_7_x PASSED      [ 82%]
tests/test_schemas.py::TestSchemas::test_query_downgrade_8_x PASSED      [ 82%]
tests/test_schemas.py::TestSchemas::test_threshold_downgrade_7_x PASSED  [ 83%]
tests/test_schemas.py::TestSchemas::test_threshold_downgrade_8_x PASSED  [ 84%]
tests/test_schemas.py::TestSchemas::test_versioned_downgrade_7_x PASSED  [ 85%]
tests/test_schemas.py::TestSchemas::test_versioned_downgrade_8_x PASSED  [ 85%]
tests/test_schemas.py::TestVersionLockSchema::test_version_lock_has_nested_previous PASSED [ 86%]
tests/test_schemas.py::TestVersionLockSchema::test_version_lock_no_previous PASSED [ 87%]
tests/test_schemas.py::TestVersions::test_stack_schema_map PASSED        [ 87%]
tests/test_specific_rules.py::TestEndpointQuery::test_os_and_platform_in_query SKIPPED [ 88%]
tests/test_specific_rules.py::TestNewTerms::test_history_window_start SKIPPED [ 89%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_field_exists SKIPPED [ 90%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_fields SKIPPED [ 90%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_fields_unique SKIPPED [ 91%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_max_limit SKIPPED [ 92%]
tests/test_specific_rules.py::TestESQLRules::test_esql_queries SKIPPED   [ 92%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_formatter_deep PASSED [ 93%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_formatter_rule PASSED [ 94%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_normalization PASSED [ 95%]
tests/test_transform_fields.py::TestGuideMarkdownPlugins::test_plugin_conversion PASSED [ 95%]
tests/test_transform_fields.py::TestGuideMarkdownPlugins::test_transform_guide_markdown_plugins PASSED [ 96%]
tests/test_utils.py::TestTimeUtils::test_caching PASSED                  [ 97%]
tests/test_utils.py::TestTimeUtils::test_event_class_normalization PASSED [ 97%]
tests/test_utils.py::TestTimeUtils::test_schema_multifields PASSED       [ 98%]
tests/test_utils.py::TestTimeUtils::test_time_normalize PASSED           [ 99%]
tests/test_version_locking.py::TestVersionLock::test_previous_entries_gte_current_min_stack PASSED [100%]

=================================== FAILURES ===================================
___________________________ ParserTests.test_keyword ___________________________

self = <tests.kuery.test_parser.ParserTests testMethod=test_keyword>

    def test_keyword(self):
        schema = {
            "a.text": "text",
            "a.keyword": "keyword",
            "b": "long",
        }
    
        self.validate('a.text:hello', FieldComparison(Field("a.text"), String("hello")), schema=schema)
        self.validate('a.keyword:hello', FieldComparison(Field("a.keyword"), String("hello")), schema=schema)
    
        self.validate('a.text:"hello"', FieldComparison(Field("a.text"), String("hello")), schema=schema)
        self.validate('a.keyword:"hello"', FieldComparison(Field("a.keyword"), String("hello")), schema=schema)
    
        self.validate('a.text:1', FieldComparison(Field("a.text"), String("1")), schema=schema)
        self.validate('a.keyword:1', FieldComparison(Field("a.keyword"), String("1")), schema=schema)
    
        self.validate('a.text:"1"', FieldComparison(Field("a.text"), String("1")), schema=schema)
        self.validate('a.keyword:"1"', FieldComparison(Field("a.keyword"), String("1")), schema=schema)
    
>       self.validate('a.keyword:*hello world*', FieldComparison(Field("a.keyword"), String("1")), schema=schema)

tests/kuery/test_parser.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/kuery/test_parser.py:22: in validate
    self.assertEqual(kql.parse(source, *args, **kwargs), tree)
E   AssertionError: Field[17 chars]Field(name='a.keyword'), value=Wildcard(value='*hello world*')) != Field[17 chars]Field(name='a.keyword'), value=String(value='1'))
________________ TestValidRules.test_all_rule_queries_optimized ________________
tests/base.py:71: in setUp
    self.fail(f'Rule loader failure: \n{RULE_LOADER_FAIL_MSG}')
E   AssertionError: Rule loader failure: 
E   Error at line:2,column:99
E   Invalid syntax
E   event.category:file and event.type:change and
E     (file.name:pam_*.so or file.path:(/etc/pam.d/* or /private/etc/pam.d/* or /usr/lib64/security/*)) and
E                                                                                                     ^
---------------------------- Captured stdout setup -----------------------------
Error loading rule in /home/runner/work/detection-rules/detection-rules/rules/cross-platform/persistence_credential_access_modify_auth_module_or_config.toml
=============================== warnings summary ===============================
../../../../../opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/config/__init__.py:1223
  /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/config/__init__.py:1223: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: typeguard
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/kuery/test_parser.py::ParserTests::test_keyword - AssertionError: Field[17 chars]Field(name='a.keyword'), value=Wildcard(value='*hello world*')) != Field[17 chars]Field(name='a.keyword'), value=String(value='1'))
FAILED tests/test_all_rules.py::TestValidRules::test_all_rule_queries_optimized - AssertionError: Rule loader failure: 
Error at line:2,column:99
Invalid syntax
event.category:file and event.type:change and
  (file.name:pam_*.so or file.path:(/etc/pam.d/* or /private/etc/pam.d/* or /usr/lib64/security/*)) and
                                                                                                  ^
============= 2 failed, 83 passed, 56 skipped, 1 warning in 12.87s =============

If you run python -m detection_rules test, you should be able to test it all locally.

@stevengoossensB
Copy link
Author

stevengoossensB commented Feb 8, 2024

Not pursuing any further. Changed syntax on our end. However field:*hello world* is a valid query and should be considered as such.

@Mikaayenson
Copy link
Contributor

We may have to take a closer look. FWIW, I added your unit test without the grammar changes and the test case still passed.

self.validate('a.keyword:*hello world*', FieldComparison(Field("a.keyword"), String("1")), schema=schema)

So if you have any more details, we can try to revisit this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants