[Q&A] How send only records that meet some conditions

[icalvete]

What is a problem?

I want send to ElasticSearch just some record based in conditions.

This is my main conf.

<source>
  bind 0.0.0.0
  <parse>
    expression /<SOME CUSTOM REGEXP>/
    @type regexp
  </parse>
  port 5514
  tag main_tag
  <transport tcp>
  </transport>
  @type syslog
</source>

This is my match for what I send to local3.

<match main_tag.local3.*>
      @type stdout
      @id debug_output
</match>

And a result in /var/log/fluent/fluentd.log

2024-12-11 19:34:57.576728732 +0000 main_tag.local3.info: {"h":"10.0.0.205","x":"apache","e":"PRO","axff":"-","ah":"53.213.185.142","at":"11/Dec/2024:19:34:57 +0000","ar":"POST /phpinfo.php HTTP/1.1","as":"200","aO":"4552","aua":"check_http/v2.3 (monitoring-plugins 2.3)","av":"my.example.com","ap":"443","aT":"0"}

I would like to send just some records to ElasticSearch....

For example just only with "as" value NOT 200.
Or with "x" with value apache OR nginx
Or some other conditions I have to think on.
Or send to different places depends on that conditions.

In short, how manage conditions if is possible.

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

No response

Environment

- Fluentd version: fluentd 1.17.0 (e763c0761c44d9734b6aa374371387a2e8406522)
- Fluent Package version: fluent-package 5.1.0
- Operating system: Ubuntu 24.04.1 LTS
- Kernel version: 6.8.0-1019-aws

[bdragoiu]

You could use the https://docs.fluentd.org/output/rewrite_tag_filter to change the tag using multiple blocks. Then simply those tags and go from there.

	<main_tag.local3.*>
		@type rewrite_tag_filter
		<rule>
			key as
			pattern ^200$
			tag as200.${tag}
                        invert  true # in order to exclude
		</rule>
		<rule>
			key x
			pattern ^(apache|nginx)$
			tag $1.${tag} # regex capture group in pattern, referenced with $1
		</rule>
	</match>

       <match as200.**></match>
       <match apache.**></match>
       <match nginx.**></match>

       <match **></match> # Do not forget to match what's left in case you care about it

[icalvete]

Thanks !!!