Sagan crashes when attempting to load example data

[wrharding]

Attempting to load in fake data from Blue Team Village's Project Obsideian from DC30 results in a core dump:

Segmentation fault (core dumped)

The container is run with the following parameters:
sudo docker run -dit --name sagan-docker -v /home/wharding/Downloads/DC30/Obsidian/filebeat/:/mnt/filebeat/ sagan-docker:0.3.0

Sagan is run with the following parameters:
/var/log/sagan # sagan -u root -F /mnt/filebeat/hmail-2022.02.12.log -d engine,syslog

[*] Configuration file /usr/local/etc/sagan.yaml loaded and 741 rules loaded.
[*] There are 741 rules loaded.
[*] 2 flexbit(s) are in use.
[*] 33 xbit(s) are in use.
[*] 0 dynamic rule(s) are loaded.
[*] Named pipe/FIFO input type: Pipe
[*] Parse log for JSON        : Enabled
[*] Client Stats              : Disabled
[*] Syslog batch:             : 1
[*] PCRE JIT                  : Enabled
[*] 
[*] Sagan version 2.0.2 is firing up on default_sensor_name (cluster: default_cluster_name)
[*] 
[*] Dropping privileges! [UID: 0 GID: 0]
[*] [lockfile.c, line 116] Lock file is present,  but Sagan isn't at pid 207 (Removing stale /var/run/sagan/sagan.pid file)
[*] ---------------------------------------------------------------------------
[*] Initializing shared memory objects.
[*] ---------------------------------------------------------------------------
[*] - Counters shared object (reload)
[*] - Xbit shared object reloaded (0 xbits loaded / max: 10000).
[*] - Flexbit shared object reloaded (0 flexbits loaded / max: 10000).
[*] - Threshold shared object reloaded (0 sources loaded / max: 10000).
[*] - After shared object reloaded (0 sources loaded / max: 10000).
[*] 
[*]  ,-._,-. 	-*> Sagan! <*-
[*]  \/)"(\/	Version 2.0.2
[*]   (_o_)	Champ Clark III & The Quadrant InfoSec Team [quadrantsec.com]
[*]   /   \/)	Copyright (C) 2009-2021 Quadrant Information Security, et al.
[*]  (|| ||) 	Using PCRE version: 8.45 2021-06-15
[*]   oo-oo
[*] 
[*] Spawning 50 Processor Threads.
[*] 
[*] Attempting to open syslog FILE (/mnt/filebeat/hmail-2022.02.12.log).
[*] Successfully opened FILE (/mnt/filebeat/hmail-2022.02.12.log) and processing events.....

The last few lines before segfault(note, I can't see the segfault thanks to a lack of dmesg utility):

[D] [processor.c, line 205] **[Parsed Syslog]*********************************
[D] [processor.c, line 206] Host: 192.168.2.1 | Program: dhcpd | Facility: local7 | Priority: info | Level: info | Tag: 192.168.2 | Date: 2022-08-25T19:12:29Z | Time: 2022-08-25T19:12:29+00:00
[D] [processor.c, line 207] Parsed message: {"fields":{"service":{"type":"hmail"}},"log":{"file":{"path":"C:\\Program Files (x86)\\hMailServer\\Logs\\hmailserver_2022-02-12.log"},"offset":763597},"input":{"type":"log"},"@timestamp":"2022-02-12T18:55:28.707Z","message":"\"SMTPD\"\t2800\t3168\t\"2022-02-12 18:55:26.820\"\t\"172.16.50.130\"\t\"SENT: 221 goodbye\"","agent":{"id":"6c9a9834-52f8-4d57-8c16-4f3bd9022475","ephemeral_id":"311bb01e-323f-46fb-8eea-3ddbabb8d60b","type":"filebeat","hostname":"files","version":"7.16.2","name":"files"},"tags":["beats_input_codec_plain_applied"],"@version":"1","host":{"name":"files"},"service":{"type":"hmail"},"ecs":{"version":"1.12.0"}}
[D] [processor.c, line 158] [batch position 0] Raw log: 192.168.2.1|local7|info|info|192.168.2.1|2022-08-25T19:12:29Z|2022-08-25T19:12:29+00:00|dhcpd|{"fields":{"service":{"type":"hmail"}},"input":{"type":"log"},"log":{"file":{"path":"C:\\Program Files (x86)\\hMailServer\\Logs\\hmailserver_2022-02-12.log"},"offset":762042},"@timestamp":"2022-02-12T18:55:28.707Z","message":"\"IMAPD\"\t2796\t3172\t\"2022-02-12 18:55:26.805\"\t\"172.16.50.130\"\t\"RECEIVED: 2 LOGOUT\"","agent":{"id":"6c9a9834-52f8-4d57-8c16-4f3bd9022475","ephemeral_id":"311bb01e-323f-46fb-8eea-3ddbabb8d60b","type":"filebeat","hostname":"files","version":"7.16.2","name":"files"},"tags":["beats_input_codec_plain_applied"],"@version":"1","host":{"name":"files"},"service":{"type":"hmail"},"ecs":{"version":"1.12.0"}}

[D] [processor.c, line 205] **[Parsed Syslog]*********************************
[D] [sagan.c, line 1225] [batch position 0] Raw log: 192.168.2.1|local7|info|info|192.168.2.1|2022-08-25T19:12:29Z|2022-08-25T19:12:29+00:00|dhcpd|{"fields":{"service":{"type":"hmail"}},"input":{"type":"log"},"log":{"file":{"path":"C:\\Program Files (x86)\\hMailServer\\Logs\\hmailserver_2022-02-12.log"},"offset":763783},"@timestamp":"2022-02-12T18:55:28.707Z","message":"\"SMTPD\"\t2840\t3176\t\"2022-02-12 18:55:26.836\"\t\"172.16.50.130\"\t\"RECEIVED: EHLO we-guess.mozilla.org\"","agent":{"id":"6c9a9834-52f8-4d57-8c16-4f3bd9022475","ephemeral_id":"311bb01e-323f-46fb-8eea-3ddbabb8d60b","type":"filebeat","hostname":"files","version":"7.16.2","name":"files"},"tags":["beats_input_codec_plain_applied"],"@version":"1","host":{"name":"files"},"service":{"type":"hmail"},"ecs":{"version":"1.12.0"}}

[D] [sagan.c, line 1225] [batch position 0] Raw log: 192.168.2.1|local7|info|info|192.168.2.1|2022-08-25T19:12:29Z|2022-08-25T19:12:29+00:00|dhcpd|{"fields":{"service":{"type":"hmail"}},"log":{"file":{"path":"C:\\Program Files (x86)\\hMailServer\\Logs\\hmailserver_2022-02-12.log"},"offset":763882},"input":{"type":"log"},"@timestamp":"2022-02-12T18:55:28.707Z","message":"\"SMTPD\"\t2840\t3176\t\"2022-02-12 18:55:26.836\"\t\"172.16.50.130\"\t\"SENT: 250-magnumtempusfinancial.com[nl]250-SIZE 20480000[nl]250-AUTH LOGIN PLAIN[nl]250 HELP\"","agent":{"id":"6c9a9834-52f8-4d57-8c16-4f3bd9022475","ephemeral_id":"311bb01e-323f-46fb-8eea-3ddbabb8d60b","type":"filebeat","hostname":"files","version":"7.16.2","name":"files"},"tags":["beats_input_codec_plain_applied"],"@version":"1","host":{"name":"files"},"service":{"type":"hmail"},"ecs":{"version":"1.12.0"}}

[D] [processor.c, line 206] Host: 192.168.2.1 | Program: dhcpd | Facility: local7 | Priority: info | Level: info | Tag: 192.168.2 | Date: 2022-08-25T19:12:29Z | Time: 2022-08-25T19:12:29+00:00
[D] [processor.c, line 207] Parsed message: {"fields":{"service":{"type":"hmail"}},"input":{"type":"log"},"log":{"file":{"path":"C:\\Program Files (x86)\\hMailServer\\Logs\\hmailserver_2022-02-12.log"},"offset":762042},"@timestamp":"2022-02-12T18:55:28.707Z","message":"\"IMAPD\"\t2796\t3172\t\"2022-02-12 18:55:26.805\"\t\"172.16.50.130\"\t\"RECEIVED: 2 LOGOUT\"","agent":{"id":"6c9a9834-52f8-4d57-8c16-4f3bd9022475","ephemeral_id":"311bb01e-323f-46fb-8eea-3ddbabb8d60b","type":"filebeat","hostname":"files","version":"7.16.2","name":"files"},"tags":["beats_input_codec_plain_applied"],"@version":"1","host":{"name":"files"},"service":{"type":"hmail"},"ecs":{"version":"1.12.0"}}
[D] [sagan.c, line 1225] [batch position 0] Raw log: 192.168.2.1|local7|info|info|192.168.2.1|2022-08-25T19:12:29Z|2022-08-25T19:12:29+00:00|dhcpd|{"fields":{"service":{"type":"hmail"}},"log":{"file":{"path":"C:\\Program Files (x86)\\hMailServer\\Logs\\hmailserver_2022-02-12.log"},"offset":764038},"input":{"type":"log"},"@timestamp":"2022-02-12T18:55:28.707Z","message":"\"SMTPD\"\t2816\t3176\t\"2022-02-12 18:55:26.836\"\t\"172.16.50.130\"\t\"RECEIVED: QUIT\"","agent":{"id":"6c9a9834-52f8-4d57-8c16-4f3bd9022475","ephemeral_id":"311bb01e-323f-46fb-8eea-3ddbabb8d60b","type":"filebeat","hostname":"files","version":"7.16.2","name":"files"},"tags":["beats_input_codec_plain_applied"],"@version":"1","host":{"name":"files"},"service":{"type":"hmail"},"ecs":{"version":"1.12.0"}}

[wrharding]

With guidance I was able to find the segfault was caused by a rule being triggered in Sagan. Note, Sagan v2.0.2 was being utilized with the CURRENT rule set and a protocol.map file from December 2021. After recompiling with the current most up to date version of Sagan a new error occurs on a smaller set of sample data (local auth.log file):

[*] 
[*] Successfully opened uncompressed file /auth.log....  processing events.....
[*] Waiting on 49/50 threads....

Thread 47 "SaganProcessor" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 844]
0x0000561583319baa in Format_JSON_Alert_EVE (Event=Event@entry=0x7f07d6b0c040, str=str@entry=0x7f07111db650 "", size=size@entry=32768) at json-handler.c:55
55	{
(gdb) bt
#0  0x0000561583319baa in Format_JSON_Alert_EVE (Event=Event@entry=0x7f07d6b0c040, str=str@entry=0x7f07111db650 "", size=size@entry=32768) at json-handler.c:55
#1  0x0000561583306e22 in Output (Event=0x7f07d6b0c040) at output.c:81
#2  0x0000561583309006 in Send_Alert (SaganProcSyslog_LOCAL=0x7f072a925720, rule_position=544, tp=..., bluedot_json=0x7f0729db0180 "", bluedot_results=<optimized out>, GeoIP_SRC=0x0, GeoIP_DEST=0x0)
    at send-alert.c:104
#3  0x000056158331fc36 in Sagan_Engine (SaganProcSyslog_LOCAL=0x7f072a925720, JSON_LOCAL=0x7f07222de8c0, dynamic_rule_flag=false) at processors/engine.c:1364
#4  0x000056158330727e in Processor () at processor.c:224
#5  0x00007f07d7cb61f5 in ?? () from /lib/ld-musl-x86_64.so.1
#6  0x0000000000000000 in ?? ()

[wrharding]

Example data:

127.0.0.1|user|debug|debug|localhost|2022-09-09T05:01:00Z|2022-09-09T01:01:00|sudo|root : TTY=pts/1 ; PWD=/ ; USER=root ; COMMAND=list
127.0.0.1|user|debug|debug|localhost|2022-09-09T05:01:00Z|2022-09-09T01:01:00|sudo|sudo: wharding : TTY=pts/5 ; PWD=/home/wharding/Projects/sagan-docker ; USER=root ; COMMAND=/bin/bash

[wrharding]

I was able to ingest upwards of 10 million lines of logs from a ~75 million log file for a segfault occurred. Note, the example data above does not contain the prerequisite space needed by Sagan. New example data should be like so:

127.0.0.1|user|debug|debug|localhost|2022-09-09T05:01:00Z|2022-09-09T01:01:00|sudo| root : TTY=pts/1 ; PWD=/ ; USER=root ; COMMAND=list
127.0.0.1|user|debug|debug|localhost|2022-09-09T05:01:00Z|2022-09-09T01:01:00|sudo| sudo: wharding : TTY=pts/5 ; PWD=/home/wharding/Projects/sagan-docker ; USER=root ; COMMAND=/bin/bash

When attempting to read in the BTV's Project Obsidian logs the syslog headers will need to be included in the JSON. See the following Sagan docs for details on what needs to be included.