Merge pull request #54 from logicmonitor/DEV-131526-Add-Proxy-support

DEV-131526 : Add proxy support
logicmonitor · Jun 21, 2023 · 4ff5ef0 · 4ff5ef0
2 parents 73de65b + 16fa1d8
commit 4ff5ef0
Show file tree

Hide file tree

Showing 6 changed files with 174 additions and 5 deletions.
diff --git a/Examples/Nomad/Nomad-fluentd.conf b/Examples/Nomad/Nomad-fluentd.conf
@@ -0,0 +1,33 @@
+<source>
+  @type forward
+  tag lm.nomad
+  port 24224
+  bind 0.0.0.0
+</source>
+
+<filter lm.**>
+    @type record_transformer
+    <record>
+        hostname "192.168.1.1"
+        tag ${tag}
+        message ${record["log"]}
+    </record>
+    remove_keys log
+</filter>
+
+<match lm.**>
+    @type copy
+    <store>
+      @type lm
+      company_name portalname
+      access_id "**************"
+      access_key "*************"
+      flush_interval 1s
+      resource_mapping {"hostname": "system.hostname"}
+      debug false
+      include_metadata true
+    </store>
+    <store>
+      @type stdout
+    </store>
+</match>
diff --git a/Examples/Nomad/README.md b/Examples/Nomad/README.md
@@ -0,0 +1,87 @@
+## Sending Nomad Logs to LogicMonitor using Fluentd
+To configure a nomad job to send log files to fluentd, you will need to use the ***fluentd*** log driver in your nomad job file. This log driver allows Nomad to send log messages to fluentd, which can then forward the logs to LogicMonitor for storage and analysis.
+
+Here is an example of how to use the ***fluentd*** log driver in your nomad job file:
+
+```hcl
+job "example-job" {
+    # ...
+    #Configure task and specify fluentd as the logging driver for the task
+    task "server" {
+        driver = "docker"
+        config {
+            # ...
+            logging {
+                type = "fluentd"
+                config {
+                    fluentd-address = "localhost:24224"
+                    labels = "custom-label,nomad-region,nomad-task-name,nomad-job-name,nomad-host-dc,nomad-host"
+                }
+            }
+            labels {
+                custom-label="example custom label"
+                nomad-region="${NOMAD_REGION}"
+                nomad-task-name="${NOMAD_TASK_NAME}"
+                nomad-job-name="${NOMAD_JOB_NAME}"
+                nomad-host-dc="${node.datacenter}"
+                nomad-host="${node.unique.name}"
+            }
+            ulimit {
+                nofile = "40960:40960"
+                }
+            # ...
+        }
+    }
+}
+```
+In this example, the ***fluentd*** log driver is used to send log messages to a fluentd instance running on ***localhost*** at port ***24224***. You will need to update the ***fluentd-address*** value to match the address of your fluentd instance.
+
+Once you have added the ***logging*** block to your job file, you can submit the job to Nomad as usual. The Nomad agent will then start sending log messages to fluentd.
+
+To include custom labels in the fluentd record, you will need to specify the ***labels*** parameter in the config block of the ***logging*** stanza in your Nomad job file. The labels parameter should be set to a list of label keys from the ***labels*** parameter specified in the config block of the ***task*** stanza, the specified labels will be added to the fluentd record for each log message.
+
+To receive logs from Nomad, you will need to configure fluentd to listen for incoming log messages from Nomad. The specific configuration will depend on your fluentd setup, but in general, you will need to add a ***source***, ***filter*** and ***match*** sections to your fluentd configuration file to listen for logs from Nomad and route them to the appropriate destination.
+
+Here is an example of a fluentd configuration that can receive logs from Nomad:
+
+```yaml
+<source>
+  @type forward
+  tag lm.nomad
+  port 24224
+  bind 0.0.0.0
+</source>
+
+<filter lm.**>
+    @type record_transformer
+    <record>
+        hostname "192.168.1.1"
+        tag ${tag}
+        message ${record["log"]}
+    </record>
+    remove_keys log
+</filter>
+
+<match lm.**>
+    @type copy
+    <store>
+      @type lm
+      company_name portalname
+      access_id "**************"
+      access_key "*************"
+      flush_interval 1s
+      resource_mapping {"hostname": "system.hostname"}
+      debug false
+      include_metadata true
+    </store>
+    <store>
+      @type stdout
+    </store>
+</match>
+```
+
+In this example, the ***source*** section configures fluentd to listen for incoming logs on all network interfaces at port ***24224***. This is the default port used by the fluentd log driver in Nomad, so you will need to make sure that this matches the ***fluentd-address value*** in your Nomad job file.
+
+The ***filter*** section is used in combination with a match section, where the match section specifies a pattern to match incoming log messages, and the filter section specifies one or more transformations to apply to the matched log messages. In the above example we are setting the message value to the value of the record log field and then removing the log value from the record key list since we no longer need it. We are also specifiying a hostname to use when performing the required resource mapping in the ***match*** section of the fluentd config so LogicMonitor knows what resource the log should be assocaited with.
+
+The ***match*** section specifies a pattern to match log messages from Nomad. In this example, the pattern ***'lm.\*\**** is used, which will match all log messages with a tag that begins with ***lm***.. The matched log messages will then be routed to the LM output plugin, which will send the logs to the specified LogicMonitor portal for ingestion.
diff --git a/Examples/Nomad/nomad-job.plan b/Examples/Nomad/nomad-job.plan
@@ -0,0 +1,41 @@
+job "example-job" {
+    datacenters = ["example-dc"]
+    group "example-group" {
+        network {
+            port "http" {
+                static = "5678"
+            }
+        }
+        task "server" {
+            driver = "docker"
+            config {
+                image = "hashicorp/http-echo"
+                logging {
+                    type = "fluentd"
+                    config {
+                        fluentd-address = "localhost:24224"
+                        labels = "custom-label,nomad-region,nomad-task-name,nomad-job-name,nomad-host-dc,nomad-host"
+                    }
+                }
+                labels {
+                    custom-label="example custom label"
+                    nomad-region="${NOMAD_REGION}"
+                    nomad-task-name="${NOMAD_TASK_NAME}"
+                    nomad-job-name="${NOMAD_JOB_NAME}"
+                    nomad-host-dc="${node.datacenter}"
+                    nomad-host="${node.unique.name}"
+                }
+                ulimit {
+                    nofile = "40960:40960"
+                    }
+                ports = ["http"]
+                args = [
+                    "-listen",
+                    ":5678",
+                    "-text",
+                    "hello world",
+                ]
+            }
+        }
+    }
+}
diff --git a/README.md b/README.md
@@ -71,6 +71,7 @@ See the [LogicMonitor Helm repository](https://github.com/logicmonitor/k8s-helm-
 | `force_encoding` | Specify charset when logs contains invalid utf-8 characters. |
 | `include_metadata` | When `true`, appends additional metadata to the log. default `false`.  |
 | `device_less_logs` | When `true`, do not map log with any resource. record must have `service` when `true`. default `false`. |
+| `http_proxy` | http proxy string eg. `http://user:pass@proxy.server:port`. Default `nil`  |
 
 
 
diff --git a/lib/fluent/plugin/out_lm.rb b/lib/fluent/plugin/out_lm.rb
@@ -46,6 +46,8 @@ class LmOutput < BufferedOutput
     config_param :version_id,  :string, :default => "version_id"
 
     config_param :device_less_logs,  :bool, :default => false
+
+    config_param :http_proxy,   :string, :default => nil
 
     # This method is called before starting.
     # 'conf' is a Hash that includes configuration parameters.
@@ -58,7 +60,13 @@ def configure(conf)
     # Open sockets or files here.
     def start
       super
-      @http_client = Net::HTTP::Persistent.new name: "fluent-plugin-lm-logs"
+      proxy_uri = :ENV
+      if @http_proxy
+        proxy_uri = URI.parse(http_proxy)
+      elsif ENV['HTTP_PROXY'] || ENV['http_proxy']
+        log.info("Using HTTP proxy defined in environment variable")
+      end
+      @http_client = Net::HTTP::Persistent.new name: "fluent-plugin-lm-logs", proxy: proxy_uri
       @http_client.override_headers["Content-Type"] = "application/json"
       @http_client.override_headers["User-Agent"] = log_source + "/" + LmLogsFluentPlugin::VERSION
       @url = "https://#{@company_name}.logicmonitor.com/rest/log/ingest"
@@ -195,9 +203,8 @@ def send_batch(events)
       end
 
       resp = @http_client.request @uri, request
-
-      if @debug || (!resp.kind_of? Net::HTTPSuccess)
-        log.info "Status code:#{resp.code} Request Id:#{resp.header['x-request-id']}"
+      if @debug || resp.kind_of?(Net::HTTPMultiStatus) || !resp.kind_of?(Net::HTTPSuccess) 
+        log.info "Status code:#{resp.code} Request Id:#{resp.header['x-request-id']} message:#{resp.body}"
       end
     end
 

diff --git a/lib/fluent/plugin/version.rb b/lib/fluent/plugin/version.rb
@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LmLogsFluentPlugin
-    VERSION = '1.0.5'
+    VERSION = '1.0.6'
 end