Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to get rid of debug log "Detected ES 8.x or above: document type will not be used" #1024

Open
1 task done
dwidyna opened this issue Jun 29, 2023 · 5 comments
Open
1 task done

Comments

@dwidyna
Copy link

dwidyna commented Jun 29, 2023

(check apply)

Problem

I was using fluentd with your plugin sending logs to Elasticsearch 5.6 . Now I am upgrading ES to 8.8.
I changed fluent/plugin configuration and mapping templates to get rid of "types" that are no longer present in ES 8.8.
I cannot get rid of this log message "[debug]: #0 [match_es_fluent_all_copy_aws] Detected ES 8.x or above: document type will not be used." (I'm getting same message regarding other tags as well).

Steps to replicate

fluent.conf

<system>
  rpc_endpoint 0.0.0.0:24444
  log_level debug
</system>

<source>
  @type forward
  @id source_forward_generic
  port 24224
</source>

<label @FLUENT_LOG>
  <filter fluent.**>
    @type record_transformer
    @id filter_record_transformer_fluent_all
    renew_record true
    enable_ruby true
    <record>
      timestamp ${time.utc.strftime('%Y-%m-%d %H:%M:%S.%6N')}
      severity ${tag_parts.last}
      message ${record.to_json}
      tag_text ${record.dig("tag")}
      tag_keyword ${record.dig("tag")}
    </record>
  </filter>

  <match>
    @type elasticsearch
    @id match_es_fluent_all_copy_aws

    @include _common_elasticsearch.conf

    #below index name is not used:
    index_name fluentd_logs_fallback

    #index data into yearly rotated index:
    logstash_format true
    logstash_prefix fluentd_logs_v_1_0_0
    logstash_prefix_separator _
    logstash_dateformat %Y

    #take time from record itself but don't index @timestamp field:
    time_key timestamp
    time_key_format "%Y-%m-%d %H:%M:%S.%N"
    time_precision 6
    include_timestamp false
    time_key_exclude_timestamp true

    #load specific index template:
    templates {
      "template_fluentd_logs_v_1_0_0": "/fluentd/etc/es_mappings/template_fluentd_logs.json"
    }
  </match>
</label>

_common_elasticsearch.conf

    #general settings:
    scheme http
    host elasticsearch
    port 9200
    user elasticsearch
    password es_password

    #Managed Elasticsearch service in AWS needs it false (shield case)
    reload_connections true

    validate_client_version true
    verify_es_version_at_startup true
    suppress_type_name true

    max_retry_putting_template 100

    <buffer>
      flush_interval 1s
    </buffer>

/fluentd/etc/es_mappings/template_fluentd_logs.json

{
    "order": 0,
    "index_patterns": [
        "fluentd_logs_v_1_0_0_*"
    ],
    "settings": {
        "index": {
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
    },
    "mappings": {
        "dynamic": "strict",
        "properties": {
            "timestamp": {
                "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
                "type": "date"
            },
            "severity": {
                "type": "keyword"
            },
            "tag_keyword": {
                "type": "keyword"
            },
            "tag_text": {
                "type": "text"
            },
            "message": {
                "type": "text"
            }
        }
    },
    "aliases": {}
}

Expected Behavior or What you need to ask

No logs regarding the type as they are polluting logs and I believe no "type" resides in my config.

Using Fluentd and ES plugin versions

  • OS version Ubuntu 22 with Docker 24
  • Environment: ES 8.8, fluentd 1.16.1 (I'm using docker image fluent/fluentd:v1.16.1-debian-amd64-1.0)
  • ES plugin 5.2.5 / 5.3.0
    • paste boot log of fluentd or td-agent
    • paste result of fluent-gem list
async (1.31.0)
async-http (0.60.1)
async-io (1.34.3)
async-pool (0.4.0)
base64 (default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (default: 3.1.1)
bundler (default: 2.3.26)
cgi (default: 0.3.6)
concurrent-ruby (1.2.2)
console (1.16.2)
cool.io (1.7.1)
csv (default: 3.2.5)
date (default: 3.2.2)
debug (1.6.3)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
drb (default: 2.1.0)
elastic-transport (8.2.1)
elasticsearch (8.8.0)
elasticsearch-api (8.8.0)
english (default: 0.7.1)
erb (default: 2.2.3)
error_highlight (default: 0.3.0)
etc (default: 1.3.0)
excon (0.100.0)
faraday (2.7.8)
faraday-excon (2.1.0)
faraday-net_http (3.0.2)
fcntl (default: 1.0.1)
fiber-local (1.0.0)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluentd (1.16.1)
forwardable (default: 1.3.2)
getoptlong (default: 0.1.1)
http_parser.rb (0.8.0)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.1)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
json (2.6.3, default: 2.6.1)
logger (default: 1.5.0)
matrix (0.4.2)
minitest (5.15.0)
msgpack (1.7.1, 1.7.0)
multi_json (1.15.0)
mutex_m (default: 0.1.1)
net-ftp (0.1.3)
net-http (default: 0.3.0)
net-imap (0.2.3)
net-pop (0.1.1)
net-protocol (default: 0.1.2)
net-smtp (0.3.1)
nio4r (2.5.9)
nkf (default: 0.1.1)
observer (default: 0.1.1)
oj (3.14.3)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
openssl (default: 3.0.1)
optparse (default: 0.2.0)
ostruct (default: 0.5.2)
pathname (default: 0.2.0)
power_assert (2.0.1)
pp (default: 0.3.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
protocol-hpack (1.4.2)
protocol-http (0.24.1)
protocol-http1 (0.15.0)
protocol-http2 (0.15.1)
pstore (default: 0.1.1)
psych (default: 4.0.4)
racc (default: 1.6.0)
rake (13.0.6)
rbs (2.7.0)
rdoc (default: 6.4.0)
readline (default: 0.0.3)
readline-ext (default: 0.1.4)
reline (default: 0.3.1)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.2.0)
serverengine (2.3.2)
set (default: 1.0.2)
shellwords (default: 0.1.0)
sigdump (0.2.4)
singleton (default: 0.1.1)
stringio (default: 3.0.1)
strptime (0.2.5)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
tempfile (default: 0.1.2)
test-unit (3.5.3)
time (default: 0.2.2)
timeout (default: 0.2.0)
timers (4.3.5)
tmpdir (default: 0.1.2)
traces (0.9.1)
tsort (default: 0.1.0)
typeprof (0.21.3)
tzinfo (2.0.6)
tzinfo-data (1.2023.3)
un (default: 0.2.0)
uri (default: 0.12.1)
weakref (default: 0.1.1)
webrick (1.8.1)
yajl-ruby (1.4.3)
yaml (default: 0.2.0)
zlib (default: 2.1.1)
  • ES version 8.8.1 (docker docker.elastic.co/elasticsearch/elasticsearch:8.8.1-amd64)
  • ES template(s) pasted above
@dwidyna
Copy link
Author

dwidyna commented Sep 5, 2023

Hello guys, any chance for solution ?

@psandeep09
Copy link

Hello, any update

@Floppe
Copy link

Floppe commented Jan 22, 2024

Some months later, any solution?

@prateekshetty-eox
Copy link

Any updates here?

1 similar comment
@harish2497
Copy link

Any updates here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants