renamed to sematext-agent-docker

Dockerfile

@@ -0,0 +1,13 @@
+FROM mhart/alpine-node:4
+RUN apk update && apk add --update procps git curl && rm -rf /var/cache/apk/*
+RUN mkdir -p /usr/src/app
+WORKDIR /usr/src/app
+
+COPY package.json /usr/src/app/
+COPY . /usr/src/app
+RUN npm install -g
+
+COPY ./run.sh /run.sh
+RUN chmod +x /run.sh
+EXPOSE 9000
+CMD /run.sh

LICENSE

@@ -1,4 +1,4 @@
-                                 Apache License
+Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/
 
@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright {yyyy} {name of copyright owner}
+   Copyright 2015 Sematext Group, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -0,0 +1,116 @@
+
+# Sematext Docker Agent
+
+This Agent collects Metrics, Events and Logs from the Docker API for [SPM Docker Monitoring](http://sematext.com/spm/integrations/docker-monitoring.html)(Application Performance Monitoring) & [Logsene / Hosted ELK](http://sematext.com/logsene) (Log Management)
+
+![](https://sematext.files.wordpress.com/2015/11/sematextdockeragent.png?w=640)
+
+Gathered information:
+- Operating System Metrics of the Host machine (CPU / Mem / Swap/ ...) 
+- Docker Container **Metrics/Stats**
+	- CPU Usage
+	- Memory Usage
+	- Network Stats
+	- Disk I/O Stats
+- **Docker Events**
+    - Version Information on Startup:
+        - server-info – created by spm-agent framework with node.js and OS version info on startup
+        - docker-info – Docker Version, API Version, Kernel Version on startup
+    - Docker Events:
+        - Container Lifecycle Events like
+            - create, exec_create, destroy, export
+        - Container Runtime Events like
+            - die, exec_start, kill, pause, restart, start, stop, unpause, ...
+- Docker **Logs**
+  - default fields
+	- hostname / IP address
+	- container id
+	- container name
+	- image name
+	- message
+  - Log format detection and log parsers: 
+	- NGINX
+	- APACHE httpd, Kafka, Solr, HBase, Zookeeper, Cassandra
+	- MySQL
+	- MongoDB
+	- Redis
+	- Elasticsearch
+	- Nsq.io
+	- JSON, ... 
+
+## Installation 
+1. Get a free account at [sematext.com/spm](https://apps.sematext.com/users-web/register.do)  
+2. [Create an SPM App](https://apps.sematext.com/spm-reports/registerApplication.do) of type "Docker" and copy the SPM Application Token 
+   - For logs (optional) [create a Logsene App](https://apps.sematext.com/logsene-reports/registerApplication.do) to an App Token for [Logsene](http://www.sematext.com/logsene/)  
+3. Run the image 
+	```
+	docker pull sematext/sematext-agent-agent
+	docker run -d --name sematext-agent-docker -e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN  -e HOSTNAME=$HOSTNAME  -v /var/run/docker.sock:/var/run/docker.sock sematext/sematext-agent-docker
+	```
+
+	**Required Parameters:**
+	- -e SPM_TOKEN - SPM Application Token
+	- -e HOSTNAME - Name of the docker host e.g. '$HOSTNAME' for Amazon ECS see HOSTNAME_LOOKUP_URL 
+	- -v /var/run/docker.sock - Path to the docker socket
+
+	**Optional Parameters:**
+	- --privileged  might be required for Security Enhanced Linux (the better way is to have the right policy ...)
+	- -e HOSTNAME_LOOKUP_URL - On Amazon ECS, a [metadata query](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) must be used to get the instance hostname (e.g. "169.254.169.254/latest/meta-data/local-hostname")
+	- -e HTTPS_PROXY - url for a proxy server
+
+	**Docker Logs Parameters:**
+	- -e LOGSENE_TOKEN - Logsene Application Token for logs
+	- -e REMOVE_ANSI_ESCAPE_SEQ=enabled - removes e.g. ANSI Terminal color codes from logs for pattern matching 
+	- Whitelist containers for logging 
+	  - -e MATCH_BY_NAME - A regular expression to white list container names 
+	  - -e MATCH_BY_IMAGE - A regular expression to white list image names 
+	- Blacklist containers 
+	  - -e SKIP_BY_NAME - A regular expression to black list container names 
+	  - -e SKIP_BY_IMAGE - A regular expression to black list image names for logging 
+	  - -v /yourpatterns/patterns.yml:/etc/logagent/patterns.yml - to provide custom patterns for log parsing, see [logagent-js](https://github.com/sematext/logagent-js)
+
+
+	You’ll see your Docker metrics in SPM after about a minute.
+
+5. Watch metrics, use anomaly detection for alerts, create e-mail reports and [much more ...](http://blog.sematext.com/2015/06/09/docker-monitoring-support/)
+
+![](https://sematext.files.wordpress.com/2015/06/docker-overview-2.png)
+
+![](https://sematext.files.wordpress.com/2015/06/docker-network-metrics.png)
+
+Docker Events:
+![](https://sematext.files.wordpress.com/2015/06/bildschirmfoto-2015-06-24-um-13-56-39.png)
+
+## Integrated Log Parser
+
+SPM for Docker recognizes log formats - so your logs arrive in a structured format in Logsene!
+The format recognition, data extractions, date parsing etc. is provided by [logagent-js](https://github.com/sematext/logagent-js) and covers:
+- Format detection e.g. for
+    	- Mongo DB
+	- Nginx
+	- Apache httpd, Kafka, Cassandra, HBase, Solr, Zookeeper
+	- MySQL
+	- Redis  
+- plain text log messages
+- line delimited JSON logs
+
+To use a custom pattern definition simply mount a volume to '/etc/logagent/patterns.yml':
+```
+-v /mydir/patterns.yml:/etc/logagent/patterns.yml
+```
+
+Feel free to contribute to [logagent-js](https://github.com/sematext/logagent-js) to enrich the default pattern set. 
+
+# Installation on CoreOS Linux
+
+SPM for Docker can monitor CoreOS clusters including metrics and logs from Docker and journald.   
+See: [Setup SPM on CoreOS](https://github.com/sematext/sematext-agent-docker/tree/master/coreos)
+
+# Support
+
+1. Please check the [SPM for Docker Wiki](https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+for+Docker)
+2. If you have questions about SPM for Docker, chat with us in the [SPM user interface](https://apps.sematext.com/users-web/login.do) or drop an e-mail to support@sematext.com
+3. Open an issue [here](https://github.com/sematext/sematext-agent-docker/issues) 
+4. Contribution guide [here](https://github.com/sematext/sematext-agent-docker/blob/master/contribute.md)
+
+

bin/spm-client-diagnostics.js

@@ -0,0 +1,48 @@
+#!/usr/bin/env node
+/*
+ * @copyright Copyright (c) Sematext Group, Inc. - All Rights Reserved
+ *
+ * @licence SPM for Docker is free-to-use, proprietary software.
+ * THIS IS PROPRIETARY SOURCE CODE OF Sematext Group, Inc. (Sematext)
+ * This source code may not be copied, reverse engineered, or altered for any purpose.
+ * This source code is to be used exclusively by users and customers of Sematext.
+ * Please see the full license (found in LICENSE in this distribution) for details on its license and the licenses of its dependencies.
+ */
+var fs = require('fs')
+var AdmZip = require('adm-zip')
+var zip = new AdmZip()
+var config = require('spm-agent').Config
+var util = require('util')
+var ls = require('ls')
+var os = require('os')
+var path = require('path')
+
+var dockerInfo = {}
+
+try {
+	dockerInfo = fs.statSync ('/var/run/docker.sock')
+} catch (ex) {
+	dockerInfo = ex
+}
+var systemInfo = {
+  operatingSystem: os.type() + ', ' + os.platform() + ', ' + os.release() + ', ' + os.arch(),
+  processVersions: process.versions,
+  processEnvironment: process.env,
+  dockerSocketInfo: dockerInfo
+}
+
+var cfgDumpFileName = path.join(os.tmpdir(), 'spm-cfg-dump.txt')
+fs.writeFileSync(cfgDumpFileName, util.inspect(config).toString() + '\nSystem-Info:\n' + util.inspect(systemInfo))
+console.log(util.inspect(config).toString() + '\nSystem-Info:\n' + util.inspect(systemInfo))
+var logfiles = ls(config.logger.dir + '/*')
+zip.addLocalFile(cfgDumpFileName)
+console.log ('Adding file ' + cfgDumpFileName)
+logfiles.forEach(function (f) {
+  console.log ('Adding file ' + f.file )
+  zip.addLocalFile(f.full)
+})
+var archFileName = os.tmpdir() + 'spm-diagnose.zip'
+zip.writeZip(archFileName)
+console.log('SPM diagnostics info is in  ' + archFileName)
+console.log('Please e-mail the file to spm-support@sematext.com')
+fs.unlink(cfgDumpFileName, function () {})

contribute.md

@@ -0,0 +1,69 @@
+# Contributing
+
+First off, thanks for taking the time to contribute! 
+
+We encourage users to open issues, because it's impossible to test everything on all platforms - so please let us know if something goes wrong or if you need enhancements. 
+
+If you are developer and like to contribute to this repository, please fork it and create a pull request. 
+
+The following section provides information to setup the test environment (assuming you know already git, node.js and Docker).
+
+## Building your own sematext-agent-docker image 
+
+SPM for Docker is implemented in node.js and this package provides an executable "sematext-agent-docker".
+It could run directly on the Docker host, for example to test a new version during development. 
+We like to make deployment easy and wrap node.js (actually io.js) and the scripts from this repository into a docker image (see Dockerfile and run.sh in this repository) - published on docker hub labeled as [sematext/sematext-agent-docker](https://registry.hub.docker.com/u/sematext/sematext-agent-docker/).
+
+To install sematext-agent-docker use
+```
+npm i sematext/sematext-agent-docker -g 
+```
+Please note this module is not published on npm, because it should be used with the docker image. 
+
+## Running the node.js agent 
+
+```
+sematext-agent-docker YOUR-SPM-APP-TOKEN-HERE
+```
+
+## Permissions and security
+
+sematext-agent-docker needs access to the docker unix socket (default: /var/run/docker.sock).
+
+```
+sudo sematext-agent-docker YOUR-SPM-APP-TOKEN-HERE
+```
+
+We recommend to run spm-docker on a user account, belonging to the 'docker' group. 
+Please refer to the instructions for your operating system, e.g.:
+The docker install instructions for Debian: https://docs.docker.com/installation/debian/
+
+```
+# Add the docker group if it doesn't already exist.
+$ sudo groupadd docker
+
+# Add the connected user "${USER}" to the docker group.
+# Change the user name to match your preferred user.
+# You may have to logout and log back in again for
+# this to take effect.
+$ sudo gpasswd -a ${USER} docker
+
+# Restart the Docker daemon.
+$ sudo service docker restart
+```
+
+## Build the docker image from sources
+
+The source directory contains the [Dockerfile](https://github.com/sematext/sematext-agent-docker/blob/master/Dockerfile) and the runner script [run.sh](https://github.com/sematext/sematext-agent-docker/blob/master/run.sh)
+
+```
+sudo docker build -t sematext/sematext-agent-docker-local .
+```
+
+## Running SPM Agent for Docker as docker container
+
+```
+docker run  -d -e SPM_TOKEN=76349b1d-XXXX-XXXX-XXXX-812f0fe85699 -e HOSTNAME=$HOSTNAME -v /var/run/docker.sock:/var/run/docker.sock sematext/sematext-agent-docker-local
+```
+
+

coreos/README.md

@@ -0,0 +1,85 @@
+# Setup SPM on CoreOS
+
+This directory provides fleet units to install [SPM](http://sematext.com/spm/integrations/coreos-monitoring.html) on CoreOS
+- [sematext-agent.service](https://github.com/sematext/sematext-agent-docker/blob/master/coreos/sematext-agent.service)
+
+	- sematext-agent.service starts Sematext Agent for Docker on all hosts 
+	- It takes the SPM and Logsene App Tokens and the TCP port for the logging gateway from etcd
+	- It starts on every CoreOS host (global unit)
+
+- [logsene.service](https://github.com/sematext/sematext-agent-docker/blob/master/coreos/spm-agent.service)
+
+	- It forwards logs from journald to the logging gateway running as part of sematext-agent-docker
+	- All fields stored in the journal are then available in [Logsene](http://www.sematext.com/logsene/index.html)
+
+- [cloud-config.example.yml](https://github.com/sematext/sematext-agent-docker/blob/master/coreos/cloud-config.example.yml) an example, which integrates spm-agent.service and logsene.service in a cloud config file for deployments on new cluster nodes
+
+Read more about [Centralized Log Management and Monitoring for CoreOS Clusters](http://blog.sematext.com/2015/07/21/coreos-clusters-centralized-log-management-and-monitoring/)  
+
+
+# Quickstart - get up and running in 5 minutes
+
+1. Get a free account [apps.sematext.com](https://apps.sematext.com/users-web/register.do)  
+2. [Create an SPM App of type “Docker”](https://apps.sematext.com/spm-reports/registerApplication.do) to obtain the SPM Application Token
+3. Create a [Logsene](http://www.sematext.com/logsene/) App to obtain the Logsene Token
+4. Run the [install script](https://github.com/sematext/sematext-agent-docker/blob/master/coreos/install_agent.sh) on one of your CoreOS machines - services will be distributed to all machines via fleet
+
+```
+wget https://raw.githubusercontent.com/sematext/sematext-agent-docker/master/coreos/install_agent.sh -O install_agent.sh
+chmod +x install_agent.sh
+./install_agent.sh YOUR_SPM_TOKEN YOUR_LOGSENE_TOKEN 9000
+```
+
+Parameters for install_agent.sh:
+
+1. SPM Token
+2. Logsene Token
+3. Port for the Logging Gateway (exposed TCP port in spm-agent) e.g. 9000
+
+# Installation Step by Step - understand what's going on ...
+
+### Preparation:
+
+1. Get a free account [apps.sematext.com](https://apps.sematext.com/users-web/register.do)  
+2. [Create an SPM App of type “Docker”](https://apps.sematext.com/spm-reports/registerApplication.do) to obtain the SPM Application Token
+3. Create a [Logsene](http://www.sematext.com/logsene/) App to obtain the Logsene Token
+4. Store the configuration in etcd, the Logsene Gateway Port is 9000 by default. 
+
+```
+etcdctl set /sematext.com/myapp/spm/token SPM_TOKEN
+etcdctl set /sematext.com/myapp/logsene/token LOGSENE_TOKEN
+etcdctl set /sematext.com/myapp/logsene/gateway_port LOGSENE_GATEWAY_PORT
+```
+
+5. Download the service files and install it with fleet
+
+```
+# INSTALLATION
+# Download the unit file for SPM
+wget https://raw.githubusercontent.com/sematext/sematext-agent-docker/master/coreos/sematext-agent.service -O sematext-agent.service
+# Start SPM Agent in the whole cluster
+fleetctl load sematext-agent.service
+fleetctl start sematext-agent.service
+# Download the unit file for Logsene
+wget https://raw.githubusercontent.com/sematext/sematext-agent-docker/master/coreos/logsene.service -O logsene.service
+# Start the log forwarding service
+fleetctl load logsene.service
+fleetctl start logsene.service
+```
+
+# Installation using cloud config
+
+An example is provided here [cloud-config.example.yml](https://github.com/sematext/sematext-agent-docker/blob/master/coreos/cloud-config.example.yml). Please don't forget to set the App Tokens in etcd!
+
+```
+etcdctl set /sematext.com/myapp/spm/token SPM_TOKEN
+etcdctl set /sematext.com/myapp/logsene/token LOGSENE_TOKEN
+etcdctl set /sematext.com/myapp/logsene/gateway_port LOGSENE_GATEWAY_PORT
+```
+
+# Contributions are welcome
+
+If you see a way to improve the setup, make things easier or discovered a bug - please submit a pull request.  
+
+
+