Skip to content

Commit efedb4c

Browse files
mjwolfmjwolf
authored
[citrix_adc] Support parsing syslog RFC 5424 messages (elastic#12608)
With citrix_adc, add support for parsing syslog RFC5424 formatted messages. Citrix ADC can send these messages with RFC 5424 compliance mode is enabled in the appliance. This adds support for these messages in the ingest pipeline, along with some test messages formatted in RFC 5424.
1 parent bffb404 commit efedb4c

File tree

7 files changed

+317
-3
lines changed

7 files changed

+317
-3
lines changed

‎packages/citrix_adc/_dev/build/docs/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ For step-by-step instructions on how to set up an integration, see the [Getting
7979

8080
The Citrix WAF GUI can be used to configure syslog servers and WAF message types to be sent to the syslog servers. Refer to [How to Send Application Firewall Messages to a Separate Syslog Server](https://support.citrix.com/s/article/CTX138973-how-to-send-application-firewall-messages-to-a-separate-syslog-server) and [How to Send NetScaler Application Firewall Logs to Syslog Server and NS.log](https://support.citrix.com/s/article/CTX483235-send-logs-to-external-syslog-server?language=en_US) for details.
8181

82+
**Note:** It is recommended to use RFC 5424 compliant syslog messages, if supported by NetScaler. Support for RFC 5424 was added in NetScaler 14.1. Refer to [Configuring audit log action](https://docs.netscaler.com/en-us/citrix-adc/current-release/system/audit-logging/configuring-audit-logging.html#configuring-audit-log-action).
83+
8284
## Validation
8385

8486
After the integration is successfully configured, clicking on the Assets tab of the Citrix ADC Integration should display a list of available dashboards. Click on the dashboard available for your configured datastream. It should be populated with the required data.
@@ -184,4 +186,4 @@ The `citrix_adc.log` dataset provides events from the configured syslog server.
184186

185187
Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields.
186188

187-
{{fields "log"}}
189+
{{fields "log"}}

‎packages/citrix_adc/changelog.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "1.15.0"
3+
changes:
4+
- description: "Add support for parsing RFC5424 syslog messages"
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/12608
27
- version: "1.14.0"
38
changes:
49
- description: "Update grok lines for discrepancies seen in the wild, and remove newlines to fix sporadic weird errors"

‎packages/citrix_adc/data_stream/log/_dev/test/pipeline/test-citrix-rfc5424.json

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{
2+
"events": [
3+
{
4+
"@timestamp": "2025-01-12T21:06:37Z",
5+
"message": "<131>1 2025-01-12T21:06:37Z MY-CITRIX-HOST ICA 0-PPE-0 - - default Message 4357642 0 : \"ns_vpn_csg.c:17988 [TECHSUPPORT][LAUNCH][Remote ip = 175.16.199.1:46516] Message = Failed to parse CGP bind request from client\""
6+
},
7+
{
8+
"@timestamp": "2025-01-30T12:00:00Z",
9+
"message": "<173>1 2025-01-30T12:00:00Z MY-CITRIX-HOST newsyslog 79382 - - logfile turned over due to size>100K"
10+
},
11+
{
12+
"@timestamp": "2025-01-08T13:30:00Z" ,
13+
"message": "<171>1 2025-01-08T13:30:00Z MY-CITRIX-HOST httpd::authz_core 38137 - - [client 175.16.199.1:36664] AH01630: client denied by server configuration: /netscaler/ns_gui/var "
14+
},
15+
{
16+
"@timestamp": "2025-01-10T02:10:52Z",
17+
"message": "<139>1 2025-01-10T02:10:52Z MY-CITRIX-HOST [993] - - - (0-0) extract_ldap_attribute While retrieving ldap attributes mail attribute not found for x5wsp6"
18+
},
19+
{
20+
"@timestamp": "2025-01-09T14:38:06Z",
21+
"message": "<142>1 2025-01-09T14:38:06Z MY-CITRIX-HOST [993] - - - (0-0) start_ldap_auth Starting LDAP auth"
22+
},
23+
{
24+
"@timestamp": "2025-01-09T13:17:03Z",
25+
"message": "<173>1 2025-01-09T13:17:03Z MY-CITRIX-HOST httpd 22442 - - 175.16.199.1 \"HEAD /logon/LogonPoint/tmindex.html HTTP/1.1\" 200 - \"-\" \"axios/1.7.7\" \"Time: 343 microsecs\" "
26+
},
27+
{
28+
"@timestamp": "2025-01-30T10:41:20Z",
29+
"message": "<132>1 2025-01-30T10:41:20Z MY-CITRIX-HOST SSLVPN 0-PPE-0 - - default Message 11027731 0 : \"Path traversal detected |/?../../../../../../../etc/passwd|, [Source: 175.16.199.1:57123]\""
30+
}
31+
]
32+
}

‎packages/citrix_adc/data_stream/log/_dev/test/pipeline/test-citrix-rfc5424.json-expected.json

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
{
2+
"expected": [
3+
{
4+
"@timestamp": "2025-01-12T21:06:37Z",
5+
"citrix": {
6+
"cef_format": false,
7+
"detail": "<131>1 2025-01-12T21:06:37Z MY-CITRIX-HOST ICA 0-PPE-0 - - default Message 4357642 0 : \"ns_vpn_csg.c:17988 [TECHSUPPORT][LAUNCH][Remote ip = 175.16.199.1:46516] Message = Failed to parse CGP bind request from client\"",
8+
"device_event_class_id": "default",
9+
"extended": {
10+
"message": "ns_vpn_csg.c:17988 [TECHSUPPORT][LAUNCH][Remote ip = 175.16.199.1:46516] Message = Failed to parse CGP bind request from client\""
11+
},
12+
"host": "MY-CITRIX-HOST",
13+
"name": "Message"
14+
},
15+
"citrix_adc": {
16+
"log": {
17+
"message": "ns_vpn_csg.c:17988 [TECHSUPPORT][LAUNCH][Remote ip = 175.16.199.1:46516] Message = Failed to parse CGP bind request from client\"",
18+
"timestamp": "2025-01-12T21:06:37.000Z"
19+
}
20+
},
21+
"ecs": {
22+
"version": "8.11.0"
23+
},
24+
"event": {
25+
"category": [
26+
"network"
27+
],
28+
"id": "4357642",
29+
"kind": "event",
30+
"original": "<131>1 2025-01-12T21:06:37Z MY-CITRIX-HOST ICA 0-PPE-0 - - default Message 4357642 0 : \"ns_vpn_csg.c:17988 [TECHSUPPORT][LAUNCH][Remote ip = 175.16.199.1:46516] Message = Failed to parse CGP bind request from client\"",
31+
"severity": 0,
32+
"timezone": "UTC",
33+
"type": [
34+
"info"
35+
]
36+
},
37+
"observer": {
38+
"hostname": "MY-CITRIX-HOST",
39+
"product": "Netscaler",
40+
"type": "firewall",
41+
"vendor": "Citrix"
42+
},
43+
"tags": [
44+
"preserve_original_event",
45+
"preserve_duplicate_custom_fields"
46+
]
47+
},
48+
{
49+
"@timestamp": "2025-01-30T12:00:00Z",
50+
"citrix": {
51+
"cef_format": false,
52+
"detail": "<173>1 2025-01-30T12:00:00Z MY-CITRIX-HOST newsyslog 79382 - - logfile turned over due to size>100K",
53+
"extended": {
54+
"message": "logfile turned over due to size>100K"
55+
},
56+
"host": "MY-CITRIX-HOST"
57+
},
58+
"citrix_adc": {
59+
"log": {
60+
"message": "logfile turned over due to size>100K",
61+
"timestamp": "2025-01-30T12:00:00.000Z"
62+
}
63+
},
64+
"ecs": {
65+
"version": "8.11.0"
66+
},
67+
"event": {
68+
"kind": "event",
69+
"original": "<173>1 2025-01-30T12:00:00Z MY-CITRIX-HOST newsyslog 79382 - - logfile turned over due to size>100K",
70+
"timezone": "UTC"
71+
},
72+
"observer": {
73+
"hostname": "MY-CITRIX-HOST",
74+
"product": "Netscaler",
75+
"type": "firewall",
76+
"vendor": "Citrix"
77+
},
78+
"tags": [
79+
"preserve_original_event",
80+
"preserve_duplicate_custom_fields"
81+
]
82+
},
83+
{
84+
"@timestamp": "2025-01-08T13:30:00Z",
85+
"citrix": {
86+
"cef_format": false,
87+
"detail": "<171>1 2025-01-08T13:30:00Z MY-CITRIX-HOST httpd::authz_core 38137 - - [client 175.16.199.1:36664] AH01630: client denied by server configuration: /netscaler/ns_gui/var ",
88+
"extended": {
89+
"message": "[client 175.16.199.1:36664] AH01630: client denied by server configuration: /netscaler/ns_gui/var "
90+
},
91+
"host": "MY-CITRIX-HOST"
92+
},
93+
"citrix_adc": {
94+
"log": {
95+
"message": "[client 175.16.199.1:36664] AH01630: client denied by server configuration: /netscaler/ns_gui/var ",
96+
"timestamp": "2025-01-08T13:30:00.000Z"
97+
}
98+
},
99+
"ecs": {
100+
"version": "8.11.0"
101+
},
102+
"event": {
103+
"kind": "event",
104+
"original": "<171>1 2025-01-08T13:30:00Z MY-CITRIX-HOST httpd::authz_core 38137 - - [client 175.16.199.1:36664] AH01630: client denied by server configuration: /netscaler/ns_gui/var ",
105+
"timezone": "UTC"
106+
},
107+
"observer": {
108+
"hostname": "MY-CITRIX-HOST",
109+
"product": "Netscaler",
110+
"type": "firewall",
111+
"vendor": "Citrix"
112+
},
113+
"tags": [
114+
"preserve_original_event",
115+
"preserve_duplicate_custom_fields"
116+
]
117+
},
118+
{
119+
"@timestamp": "2025-01-10T02:10:52Z",
120+
"citrix": {
121+
"cef_format": false,
122+
"detail": "<139>1 2025-01-10T02:10:52Z MY-CITRIX-HOST [993] - - - (0-0) extract_ldap_attribute While retrieving ldap attributes mail attribute not found for x5wsp6",
123+
"extended": {
124+
"message": "(0-0) extract_ldap_attribute While retrieving ldap attributes mail attribute not found for x5wsp6"
125+
},
126+
"host": "MY-CITRIX-HOST"
127+
},
128+
"citrix_adc": {
129+
"log": {
130+
"message": "(0-0) extract_ldap_attribute While retrieving ldap attributes mail attribute not found for x5wsp6",
131+
"timestamp": "2025-01-10T02:10:52.000Z"
132+
}
133+
},
134+
"ecs": {
135+
"version": "8.11.0"
136+
},
137+
"event": {
138+
"kind": "event",
139+
"original": "<139>1 2025-01-10T02:10:52Z MY-CITRIX-HOST [993] - - - (0-0) extract_ldap_attribute While retrieving ldap attributes mail attribute not found for x5wsp6",
140+
"timezone": "UTC"
141+
},
142+
"observer": {
143+
"hostname": "MY-CITRIX-HOST",
144+
"product": "Netscaler",
145+
"type": "firewall",
146+
"vendor": "Citrix"
147+
},
148+
"tags": [
149+
"preserve_original_event",
150+
"preserve_duplicate_custom_fields"
151+
]
152+
},
153+
{
154+
"@timestamp": "2025-01-09T14:38:06Z",
155+
"citrix": {
156+
"cef_format": false,
157+
"detail": "<142>1 2025-01-09T14:38:06Z MY-CITRIX-HOST [993] - - - (0-0) start_ldap_auth Starting LDAP auth",
158+
"extended": {
159+
"message": "(0-0) start_ldap_auth Starting LDAP auth"
160+
},
161+
"host": "MY-CITRIX-HOST"
162+
},
163+
"citrix_adc": {
164+
"log": {
165+
"message": "(0-0) start_ldap_auth Starting LDAP auth",
166+
"timestamp": "2025-01-09T14:38:06.000Z"
167+
}
168+
},
169+
"ecs": {
170+
"version": "8.11.0"
171+
},
172+
"event": {
173+
"kind": "event",
174+
"original": "<142>1 2025-01-09T14:38:06Z MY-CITRIX-HOST [993] - - - (0-0) start_ldap_auth Starting LDAP auth",
175+
"timezone": "UTC"
176+
},
177+
"observer": {
178+
"hostname": "MY-CITRIX-HOST",
179+
"product": "Netscaler",
180+
"type": "firewall",
181+
"vendor": "Citrix"
182+
},
183+
"tags": [
184+
"preserve_original_event",
185+
"preserve_duplicate_custom_fields"
186+
]
187+
},
188+
{
189+
"@timestamp": "2025-01-09T13:17:03Z",
190+
"citrix": {
191+
"cef_format": false,
192+
"detail": "<173>1 2025-01-09T13:17:03Z MY-CITRIX-HOST httpd 22442 - - 175.16.199.1 \"HEAD /logon/LogonPoint/tmindex.html HTTP/1.1\" 200 - \"-\" \"axios/1.7.7\" \"Time: 343 microsecs\" ",
193+
"extended": {
194+
"message": "175.16.199.1 \"HEAD /logon/LogonPoint/tmindex.html HTTP/1.1\" 200 - \"-\" \"axios/1.7.7\" \"Time: 343 microsecs\" "
195+
},
196+
"host": "MY-CITRIX-HOST"
197+
},
198+
"citrix_adc": {
199+
"log": {
200+
"message": "175.16.199.1 \"HEAD /logon/LogonPoint/tmindex.html HTTP/1.1\" 200 - \"-\" \"axios/1.7.7\" \"Time: 343 microsecs\" ",
201+
"timestamp": "2025-01-09T13:17:03.000Z"
202+
}
203+
},
204+
"ecs": {
205+
"version": "8.11.0"
206+
},
207+
"event": {
208+
"kind": "event",
209+
"original": "<173>1 2025-01-09T13:17:03Z MY-CITRIX-HOST httpd 22442 - - 175.16.199.1 \"HEAD /logon/LogonPoint/tmindex.html HTTP/1.1\" 200 - \"-\" \"axios/1.7.7\" \"Time: 343 microsecs\" ",
210+
"timezone": "UTC"
211+
},
212+
"observer": {
213+
"hostname": "MY-CITRIX-HOST",
214+
"product": "Netscaler",
215+
"type": "firewall",
216+
"vendor": "Citrix"
217+
},
218+
"tags": [
219+
"preserve_original_event",
220+
"preserve_duplicate_custom_fields"
221+
]
222+
},
223+
{
224+
"@timestamp": "2025-01-30T10:41:20Z",
225+
"citrix": {
226+
"cef_format": false,
227+
"detail": "<132>1 2025-01-30T10:41:20Z MY-CITRIX-HOST SSLVPN 0-PPE-0 - - default Message 11027731 0 : \"Path traversal detected |/?../../../../../../../etc/passwd|, [Source: 175.16.199.1:57123]\"",
228+
"device_event_class_id": "default",
229+
"extended": {
230+
"message": "Path traversal detected |/?../../../../../../../etc/passwd|, [Source: 175.16.199.1:57123]\""
231+
},
232+
"host": "MY-CITRIX-HOST",
233+
"name": "Message"
234+
},
235+
"citrix_adc": {
236+
"log": {
237+
"message": "Path traversal detected |/?../../../../../../../etc/passwd|, [Source: 175.16.199.1:57123]\"",
238+
"timestamp": "2025-01-30T10:41:20.000Z"
239+
}
240+
},
241+
"ecs": {
242+
"version": "8.11.0"
243+
},
244+
"event": {
245+
"category": [
246+
"network"
247+
],
248+
"id": "11027731",
249+
"kind": "event",
250+
"original": "<132>1 2025-01-30T10:41:20Z MY-CITRIX-HOST SSLVPN 0-PPE-0 - - default Message 11027731 0 : \"Path traversal detected |/?../../../../../../../etc/passwd|, [Source: 175.16.199.1:57123]\"",
251+
"severity": 0,
252+
"timezone": "UTC",
253+
"type": [
254+
"info"
255+
]
256+
},
257+
"observer": {
258+
"hostname": "MY-CITRIX-HOST",
259+
"product": "Netscaler",
260+
"type": "firewall",
261+
"vendor": "Citrix"
262+
},
263+
"tags": [
264+
"preserve_original_event",
265+
"preserve_duplicate_custom_fields"
266+
]
267+
}
268+
]
269+
}

‎packages/citrix_adc/data_stream/log/elasticsearch/ingest_pipeline/native.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,12 @@ processors:
1313
- '^%{SPACE}%{HEADER_NOTIMEZONE} : %{DATA:_tmp.details} : +"%{GREEDYDATA:citrix.extended.message}"'
1414
- '^%{SPACE}%{HEADER} : %{DATA:_tmp.details} : +"%{GREEDYDATA:citrix.extended.message}"'
1515
- '^%{SPACE}%{HEADER} : %{DATA:_tmp.details} : +%{GREEDYDATA:citrix.extended.message}'
16+
- '^<%{NUMBER}>%{NUMBER} (%{TIMESTAMP_ISO8601:_tmp.timestamp}|-) (%{SYSLOGHOST:citrix.host}|-) (%{DATA:_tmp.appname}|-) (%{DATA:_tmp.procid}|-) (%{DATA:_tmp.msgid}|-) (%{DATA:_tmp.structured_data}|-) (%{DATA:_tmp.details} :)?%{SPACE}"?%{GREEDYDATA:citrix.extended.message}"?$'
17+
1618
pattern_definitions:
1719
HEADER_NOTIMEZONE: '(?:<%{NUMBER}>%{SPACE})?%{NATIVE_TIMESTAMP:_tmp.timestamp_native} (?:%{SYSLOGHOST:citrix.host} )?%{INT}-PPE-%{INT}'
1820
HEADER: '(?:<%{NUMBER}>%{SPACE})?%{NATIVE_TIMESTAMP:_tmp.timestamp_native} %{WORD:event.timezone}? (?:%{SYSLOGHOST:citrix.host} )?%{INT}-PPE-%{INT}'
19-
NATIVE_TIMESTAMP: '(?:%{MONTHNUM}/%{MONTHDAY}/%{YEAR}|%{YEAR}/%{MONTHNUM}/%{MONTHDAY}|%{MONTHDAY}/%{MONTHNUM}/%{YEAR}):%{HOUR}:%{MINUTE}:%{SECOND}'
21+
NATIVE_TIMESTAMP: '(?:%{MONTHNUM}/%{MONTHDAY}/%{YEAR}|%{YEAR}/%{MONTHNUM}/%{MONTHDAY}|%{MONTHDAY}/%{MONTHNUM}/%{YEAR}):%{HOUR}:%{MINUTE}:%{SECOND}'
2022
- grok:
2123
description: Parse out details.
2224
tag: grok_tmp_details
@@ -25,6 +27,7 @@ processors:
2527
- '^%{DEFAULT:_tmp.default}?%{WORD:citrix.device_event_class_id} %{GREEDYDATA:citrix.name} %{INT:event.id} %{INT:event.severity}$'
2628
pattern_definitions:
2729
DEFAULT: 'default '
30+
ignore_missing: true
2831
- set:
2932
field: event.category
3033
tag: set_event_category_network

‎packages/citrix_adc/docs/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ For step-by-step instructions on how to set up an integration, see the [Getting
7979

8080
The Citrix WAF GUI can be used to configure syslog servers and WAF message types to be sent to the syslog servers. Refer to [How to Send Application Firewall Messages to a Separate Syslog Server](https://support.citrix.com/s/article/CTX138973-how-to-send-application-firewall-messages-to-a-separate-syslog-server) and [How to Send NetScaler Application Firewall Logs to Syslog Server and NS.log](https://support.citrix.com/s/article/CTX483235-send-logs-to-external-syslog-server?language=en_US) for details.
8181

82+
**Note:** It is recommended to use RFC 5424 compliant syslog messages, if supported by NetScaler. Support for RFC 5424 was added in NetScaler 14.1. Refer to [Configuring audit log action](https://docs.netscaler.com/en-us/citrix-adc/current-release/system/audit-logging/configuring-audit-logging.html#configuring-audit-log-action).
83+
8284
## Validation
8385

8486
After the integration is successfully configured, clicking on the Assets tab of the Citrix ADC Integration should display a list of available dashboards. Click on the dashboard available for your configured datastream. It should be populated with the required data.
@@ -1471,3 +1473,4 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
14711473
| input.type | Input type. | keyword |
14721474
| log.offset | Offset of the entry in the log file. | long |
14731475
| log.source.address | Source address from which the log event was read / sent from. | keyword |
1476+

‎packages/citrix_adc/manifest.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
format_version: "3.0.2"
22
name: citrix_adc
33
title: Citrix ADC
4-
version: "1.14.0"
4+
version: "1.15.0"
55
description: This Elastic integration collects logs and metrics from Citrix ADC product.
66
type: integration
77
categories:

0 commit comments

Comments
 (0)