Clickhouse Logs to Loki

Clickhouse Log Processing

The following recipe will parse the default clickhouse-server.log multi-line format and output loki formatted and tagged logs

Requirements

• Node 14.x or higher
• cloki or Grafana Loki
• Clickhouse Server (or cluster)
• Logs of logs!

Setup

Install paStash and any required plugin using npm:

# npm install -g @pastash/pastash @pastash/output_loki

Recipe

Deploy your recipe configuration anywhere on your filesystem. Extracted tags can be customized by modifying the regex filter.

Output can be either Loki or cLoki

input {
  file {
    path => "/var/log/clickhouse-server/clickhouse-server.log"
  }
}

filter {
  multiline {
    start_line_regex => /^\d{4}.\d{2}.\d{2}/
  }
  regex {
    regex => /(\d{4}[.]\d{2}[.]\d{2}[ ]\d{2}[:]\d{2}[:]\d{2}\.\d{1,6}) \[ (\d.*) \] \{.*\} \<(.*)\> (.*?):\s(?:(.*): Code: (\d{2}))?/
    fields => ['@timestamp',pid,level,function,code]
    regex_flags => m
  }
  omit {
    blacklist => ['@timestamp','@version','path','host']
  }
  compute_field {
    field => "type"
    value => "clickhouse"
  }
}

output {
  loki {
    basic_auth_password => "some_very_secure_password_hash_here"
    basic_auth_user => "1234"
    host => "logs-us-west1.grafana.net"
    port => 80
    path => "/api/prom/push"
  }
}

Regex

The regex parser can be adapted and tested using regex101

Run

Start paStash using your recipe:

# pastash --config_file=/path/to/pastash.conf

Grafana

Use Grafana Explore to access your data through the Loki API