document_parsing_exception

[blindshare]

What is a problem?

So we're using Fluent-Bit/FluentD to gather logfiles from K8s Microservices and send them to Elastic-Search.

Lately we noticed that data is missing due to this error we found within the fluend messages:
"message":"dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=\"400 - Rejected by Elasticsearch [error type]: document_parsing_exception [reason]: '[1:756] failed to parse field [REFERENZ] of type [float] in document with id '3x4RkZmb7VeKdVaq8fDv'. Preview of field's value: '9b807de7-b228-11ef-940c-cf6318q66f78''\"

So all kinds of Microservices forward their logs to Elastic Search through fluentd and I was told that there's an "auto detect" of the field-type. But it gets only detected through the first item. So if the field "REFERENZ" had been determined as float, further fields will also be treated as floats although there may be a String (as seen in the example above). You'll have to wait until the whole "block to parse" has finished and then a new "auto type detection" will take place.

So what we want is to tell fluentd to treat "All Fields" as being of type: String ! ... then push it further to elastic-search
But what how can I configure this ?

Describe the configuration of Fluentd

I can't post the complete config here ... what parts would be needed ?

Describe the logs of Fluentd

Log entry: see above

Environment

- Fluentd version: 0.9.6
- K8s Version 1.29.8
- Operating system:Debian GNU/Linux 12 (bookworm)
- Kernel version: 6.1.0-22-amd64

[bdragoiu]

This looks like an Elasticsearch issue to me. The Preview of field's value: '9b807de7-b228-11ef-940c-cf6318q66f78' already suggests that the field is string, but ES is rejecting since it's expecting float.

You should be able to confirm this for yourself with something like this:

	<match **>
		@type copy

		<store>
			@type elasticsearch
			hosts ...
			user ...
		</store>

		<store>
			@type stdout
		</store>

	</match>

This would show you what's being submitted towards ES. You should see a JSON and see that Referenz is quoted there.