OpenCanary update, resolves multiple issues #13026

colin-stubbs · 2025-03-08T06:35:48Z

Bug
Enhancement

Proposed commit message

Resolves issues #12911, #13024, #13025. Relevant to resolution of #2518.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Resolves multiple issues:

Testing:

elastic-package lint && check && build
elastic-package test system --generate
elastic-package test pipeline --generate
elastic-package test
Manual deploy on local elastic-package managed stack and ingest of logfile
Manual deploy on remote Elastic Cloud stack and ingest of webhooks from real opencanary honeypots

How to test this PR locally

Install and operate OpenCanary, ideally via docker.
Use Elastic Agent to ingest OpenCanary log file OR webhooks.
Scan OpenCanary with nmap with scripting to trigger events, e.g. nmap -sC 127.0.0.1
Review

Related issues

Screenshots

New basic summary dashboard added,

resolves issue elastic#12911

add system tests for both

fixes based on elastic#13025

packages/opencanary/data_stream/events/_dev/test/pipeline/test-events.log

rebuild package and retest completed OK

packages/opencanary/changelog.yml

efd6 · 2025-03-10T22:22:09Z

packages/opencanary/data_stream/events/_dev/test/pipeline/test-webhooks.json

+            "event": {
+                "original": "{\"message\": \"{\\\"dst_host\\\": \\\"1.128.0.10\\\", \\\"dst_port\\\": 22, \\\"local_time\\\": \\\"2024-04-04 17:03:52.569759\\\", \\\"local_time_adjusted\\\": \\\"2024-04-04 11:03:52.569785\\\", \\\"logdata\\\": {\\\"LOCALVERSION\\\": \\\"SSH-2.0-OpenSSH_5.1p1 Debian-4\\\", \\\"PASSWORD\\\": \\\"P@ssw0rd!\\\", \\\"REMOTEVERSION\\\": \\\"SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6\\\", \\\"USERNAME\\\": \\\"jdoe\\\"}, \\\"logtype\\\": 4002, \\\"node_id\\\": \\\"opencanary-1\\\", \\\"src_host\\\": \\\"1.128.0.10\\\", \\\"src_port\\\": 11851, \\\"utc_time\\\": \\\"2024-04-04 17:03:52.569781\\\"}\"}"
+            },
+            "json": {
+                "message": "{\"dst_host\": \"1.128.0.10\", \"dst_port\": 22, \"local_time\": \"2024-04-04 17:03:52.569759\", \"local_time_adjusted\": \"2024-04-04 11:03:52.569785\", \"logdata\": {\"LOCALVERSION\": \"SSH-2.0-OpenSSH_5.1p1 Debian-4\", \"PASSWORD\": \"P@ssw0rd!\", \"REMOTEVERSION\": \"SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6\", \"USERNAME\": \"jdoe\"}, \"logtype\": 4002, \"node_id\": \"opencanary-1\", \"src_host\": \"1.128.0.10\", \"src_port\": 11851, \"utc_time\": \"2024-04-04 17:03:52.569781\"}"


Why are we sending both event and json.message?

Because that's what http_endpoint does for JSON payloads when preserve_original_event: true is set...

In the case of OpenCanary's native webhooks capability and minimal/default exampled config, it's JSON payload is literally {"message":"{\"another\":\"json text string that contains the real event\"}}"

I suspect it's internal code is really just reading the text file it produces and reading it line by line as message in it's internal event representation then encoding that as JSON and sending it off to the given URL... very similar to how fluentd works if you don't specifically instruct it to parse the line as JSON or send the actual message content only.

OK, thanks.

packages/opencanary/data_stream/events/_dev/test/pipeline/test-events.log

efd6 · 2025-03-10T22:25:22Z

packages/opencanary/data_stream/events/agent/stream/filestream.yml.hbs

+- add_host_metadata: ~
+{{#if processors}}
+{{processors}}
+{{/if}}


Add final new line.

So we've been here before @efd6 ...

This does not affect functionality in any way shape or form.

This does not affect viewing/editing functionality for anyone.

This does not violate yaml or the handlebars template standard/definition.

elastic-package does not identify this as an issue.

If this kind of thing is actually a problem elastic-package should identify it, and ideally fix it automatically via elastic-package format.

What's the actual problem here?
How are you even identifying these?

The problem is noise in the git log. For your points, I agree with 3 and 4. 1 and 2 depend on the definition of "functionality" and I don't agree when I use my definition of that (which includes maintenance, not just whether the code works).

You can see the absence marked in git logs and in the GH rendering of diffs (e.g. here for this particular case — showing up as a red circle with a red dash in the middle). It can also be seen in an editor by virtue of the absence of a new line in the rendered text (when I'm reviewing code I'll often pull the branch and experiment with changes, so I see them then).

I agree, this kind of thing should be handled by either ep or (as is the case with beats) by using an .editorconfig file.

Sure... but even elastic-package generates files like this... you can see it with sample event.json files and even the Kibana assets...

The entire integrations repo will be literally littered with files like this...

Yeap... in terms of just the yaml handlebars templates...

Quick and dirty...

user@box ~ % cat newline_test.sh #!/bin/bash printf '\n' > newline_only tail -c 1 $1 | cmp -s newline_only - || echo "$1 does not contain a final newline" # EOF user@box ~ % find ~/Documents/SRC/GitHub/elastic/integrations -type f -name \*.yml.hbs -exec ./newline_test.sh {} \; | wc -l 388 user@box ~ % find ~/Documents/SRC/GitHub/elastic/integrations -type f -name \*.yml.hbs | wc -l 1316 user@box ~ % find ~/Documents/SRC/GitHub/elastic/integrations -type f -name \*.yml.hbs -exec ./newline_test.sh {} \; | head -n 5 /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/first_epss/data_stream/vulnerability/agent/stream/cel.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/activemq/data_stream/broker/agent/stream/stream.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/activemq/data_stream/audit/agent/stream/log.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/activemq/data_stream/log/agent/stream/log.yml.hbs does not contain a final newline user@box ~ % find ~/Documents/SRC/GitHub/elastic/integrations -type f -name \*.yml.hbs -exec ./newline_test.sh {} \; | tail -n 5 /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/rubrik/data_stream/physical_hosts/agent/stream/cel.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/cisco_ios/data_stream/log/agent/stream/tcp.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/cisco_ios/data_stream/log/agent/stream/udp.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/packages/cisco_ios/data_stream/log/agent/stream/stream.yml.hbs does not contain a final newline /Users/user/Documents/SRC/GitHub/elastic/integrations/build/packages/email_report/0.0.1/data_stream/tlsrpt/agent/stream/http_endpoint.yml.hbs does not contain a final newline user@box ~ %

Sure... but even elastic-package generates files like this... you can see it with sample event.json files and even the Kibana assets...

Yes, the kibana assets are still like this, sample and expectation files are no longer rendered without a trailing new line.

efd6 · 2025-03-10T22:25:31Z

packages/opencanary/data_stream/events/agent/stream/http_endpoint.yml.hbs

+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}


Add final new line.

packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml

efd6 · 2025-03-10T22:46:19Z

packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml

+      description: Remove opencanary.logdata.MAC if MAC addresses were parsed out and mapped field preservation is not requested
      tag: remove_opencanary_logdata_MAC
-      if: ctx.source?.mac != null && ctx.destination?.mac != null
+      if: (ctx?.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields'))) && ctx.source?.mac != null && ctx.destination?.mac != null
      field: opencanary.logdata.MAC
      ignore_missing: true


Let's put these in separate processors rather than rolling them together.

packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 · 2025-03-11T06:33:07Z

/test

elasticmachine · 2025-03-11T06:46:20Z

💔 Build Failed

Buildkite Build
Commit: b1617e1

Failed CI Steps

Check integrations opencanary

History

efd6 · 2025-03-11T07:18:05Z

packages/opencanary/kibana/dashboard/opencanary-96cfb6c4-bc46-4fd4-9476-e8e9550442d9.json

+        }
+    ],
+    "type": "dashboard",
+    "typeMigrationVersion": "10.2.0",


Error: can't install the package: could not zip-install package; API status code = 422; response body = {"statusCode":422,"error":"Unprocessable Entity","message":"Document "opencanary-96cfb6c4-bc46-4fd4-9476-e8e9550442d9" belongs to a more recent version of Kibana [10.2.0] when the last known version is [8.9.0]."}

colin-stubbs added 5 commits February 28, 2025 16:19

fix alerting noise

ad8405a

resolves issue elastic#12911

migrate logfile to filestream, add http_endpoint

1df9175

add system tests for both

only remove the password field if redaction has been requested

6e1a9a9

update to 0.5.0

9aeb260

fixes based on elastic#13025

Merge remote-tracking branch 'origin/main' into opencanary

a5eaa8a

colin-stubbs requested a review from a team as a code owner March 8, 2025 06:35

Update PR link, add VNC events and improve VNC handling

e8fda68

efd6 reviewed Mar 9, 2025

View reviewed changes

packages/opencanary/data_stream/events/_dev/test/pipeline/test-events.log Show resolved Hide resolved

remove unnecessary pipeline test events

24b0ee3

rebuild package and retest completed OK

efd6 reviewed Mar 10, 2025

View reviewed changes

colin-stubbs and others added 2 commits March 11, 2025 10:48

Update packages/opencanary/changelog.yml

e38e28f

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Apply suggestions from code review

b1617e1

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 reviewed Mar 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenCanary update, resolves multiple issues #13026

OpenCanary update, resolves multiple issues #13026

colin-stubbs commented Mar 8, 2025

efd6 Mar 10, 2025

colin-stubbs Mar 11, 2025 •

edited

Loading

efd6 Mar 11, 2025

efd6 Mar 10, 2025

colin-stubbs Mar 11, 2025

efd6 Mar 11, 2025 •

edited

Loading

colin-stubbs Mar 11, 2025

colin-stubbs Mar 11, 2025

efd6 Mar 11, 2025

efd6 Mar 10, 2025

efd6 Mar 10, 2025

efd6 commented Mar 11, 2025

elasticmachine commented Mar 11, 2025 •

edited

Loading

efd6 Mar 11, 2025

OpenCanary update, resolves multiple issues #13026

Are you sure you want to change the base?

OpenCanary update, resolves multiple issues #13026

Conversation

colin-stubbs commented Mar 8, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Choose a reason for hiding this comment

colin-stubbs Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 commented Mar 11, 2025

elasticmachine commented Mar 11, 2025 • edited Loading

💔 Build Failed

Failed CI Steps

History

Choose a reason for hiding this comment

colin-stubbs Mar 11, 2025 •

edited

Loading

efd6 Mar 11, 2025 •

edited

Loading

elasticmachine commented Mar 11, 2025 •

edited

Loading