Skip to content

Commit

Permalink
Merge branch 'develop' into stable
Browse files Browse the repository at this point in the history
  • Loading branch information
driskell committed Feb 28, 2015
2 parents e2eea96 + 666ead3 commit e7423cf
Show file tree
Hide file tree
Showing 102 changed files with 7,222 additions and 5,992 deletions.
30 changes: 17 additions & 13 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
Gemfile.lock
bin
node_modules
pkg
spec/tmp
src/github.com
vendor
version.txt
.Makefile.tags
.bundle
.log-courier
.vagrant
*.gem
/.Makefile.tags
/.bundle
/.log-courier
/.vagrant
/*.gem
/Gemfile.lock
/bin
/log-courier.gemspec
/logstash-input-log-courier.gemspec
/logstash-output-log-courier.gemspec
/node_modules
/pkg
/spec/tmp
/src/github.com
/src/lc-lib/core/version.go
/vendor
/version.txt
9 changes: 6 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: prepare fix_version all log-courier gem gem_plugins push_gems test test_go test_rspec doc profile benchmark jrprofile jrbenchmark clean
.PHONY: prepare fix_version setup_root all log-courier gem gem_plugins push_gems test test_go test_rspec doc profile benchmark jrprofile jrbenchmark clean

MAKEFILE := $(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
GOPATH := $(patsubst %/,%,$(dir $(abspath $(MAKEFILE))))
Expand Down Expand Up @@ -100,9 +100,12 @@ ifneq ($(implyclean),yes)
endif

fix_version:
build/fix_version
build/fix_version "${FIX_VERSION}"

prepare: | fix_version
setup_root:
build/setup_root

prepare: | fix_version setup_root
@go version >/dev/null || (echo "Go not found. You need to install Go version 1.2-1.4: http://golang.org/doc/install"; false)
@go version | grep -q 'go version go1.[234]' || (echo "Go version 1.2-1.4, you have a version of Go that is not supported."; false)
@echo "GOPATH: $${GOPATH}"
Expand Down
181 changes: 119 additions & 62 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,79 +1,134 @@
# Log Courier [![Build Status](https://travis-ci.org/driskell/log-courier.svg?branch=develop)](https://travis-ci.org/driskell/log-courier)
# Log Courier

Log Courier is a tool created to ship log files speedily and securely to
remote [Logstash](http://logstash.net) instances for processing whilst using
small amounts of local resources. The project is an enhanced fork of
[![Build Status](https://img.shields.io/travis/driskell/log-courier/develop.svg)](https://travis-ci.org/driskell/log-courier)
[![Latest Release](https://img.shields.io/github/release/driskell/log-courier.svg)](https://github.com/driskell/log-courier/releases/latest)

Log Courier is a lightweight tool created to ship log files speedily and
securely, with low resource usage, to remote [Logstash](http://logstash.net)
instances. The project is an enhanced fork of
[Logstash Forwarder](https://github.com/elasticsearch/logstash-forwarder) 0.3.1
with many fixes and behavioural improvements.

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents** *generated with [DocToc](http://doctoc.herokuapp.com/)*

- [Features](#features)
- [Installation](#installation)
- [Requirements](#requirements)
- [Building](#building)
- [Logstash Integration](#logstash-integration)
- [Building with ZMQ support](#building-with-zmq-support)
- [Generating Certificates and Keys](#generating-certificates-and-keys)
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*

- [Main Features](#main-features)
- [Differences to Logstash Forwarder](#differences-to-logstash-forwarder)
- [Public Repositories](#public-repositories)
- [RPM](#rpm)
- [DEB](#deb)
- [Building From Source](#building-from-source)
- [Logstash Integration](#logstash-integration)
- [Generating Certificates and Keys](#generating-certificates-and-keys)
- [ZeroMQ support](#zeromq-support)
- [Documentation](#documentation)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Features

Log Courier implements the following features:
## Main Features

* Follow active log files
* Follow rotations
* Follow standard input stream
* Suspend tailing after periods of inactivity
* Set [extra fields](docs/Configuration.md#fields), supporting hashes and arrays
(`tags: ['one','two']`)
* Read events from a file or over a Unix pipeline
* Follow log file rotations and movements
* Close files after inactivity, reopening if they change
* Add [extra fields](docs/Configuration.md#fields) to events prior to shipping
* [Reload configuration](docs/Configuration.md#reloading) without restarting
* Secure TLS shipping transport with server certificate verification
* TLS client certificate verification
* Secure CurveZMQ shipping transport to load balance across multiple Logstash
instances (optional, requires ZeroMQ 4+)
* Plaintext TCP shipping transport for configuration simplicity in local
networks
* Plaintext ZMQ shipping transport
* [Administration utility](docs/AdministrationUtility.md) to monitor the
shipping speed and status
* [Multiline](docs/codecs/Multiline.md) codec
* [Filter](docs/codecs/Filter.md) codec
* Ship events securely using TLS with server (and optionally client) certificate
verification
* Ship events securely to multiple Logstash instances using ZeroMQ with Curve
security (requires ZeroMQ 4+)
* Ship events in plaintext using TCP
* Ship events in plaintext using ZeroMQ (requires ZeroMQ 3+)
* Monitor shipping speed and status with the
[Administration utility](docs/AdministrationUtility.md)
* Pre-process events using codecs (e.g. [Multiline](docs/codecs/Multiline.md),
[Filter](docs/codecs/Filter.md))
* [Logstash Integration](docs/LogstashIntegration.md) with an input and output
plugin
* Very low resource usage

## Differences to Logstash Forwarder

Log Courier is an enhanced fork of
[Logstash Forwarder](https://github.com/elasticsearch/logstash-forwarder) 0.3.1
with many fixes and behavioural improvements. The primary changes are:

* The publisher protocol is rewritten to avoid many causes of "i/o timeout"
which would result in duplicate events sent to Logstash
* The prospector and registrar are heavily revamped to handle log rotations and
movements far more reliably, and to report errors cleanly
* The harvester is improved to retry if an error occurred rather than stop
* The configuration can be reloaded without restarting
* An administration tool is available which can display the shipping speed and
status of all watched log files
* Fields configurations can contain arrays and dictionaries, not just strings
* Codec support is available which allows multiline processing at the sender
side
* A TCP transport is available which removes the requirement for SSL
certificates
* There is support for client SSL certificate verification
* Peer IP address and certificate DN can be added to received events in Logstash
to distinguish events send from different instances
* Windows: Log files are not locked allowing log rotation to occur
* Windows: Log rotation is detected correctly

## Public Repositories

### RPM

## Installation
The author maintains a **COPR** repository with RedHat/CentOS compatible RPMs
that may be installed using `yum`. This repository depends on the widely used
**EPEL** repository for dependencies.

### Requirements
The **EPEL** repository can be installed automatically on CentOS distributions
by running `yum install epel-release`. Otherwise, you may follow the
instructions on the [EPEL homepage](https://fedoraproject.org/wiki/EPEL).

1. \*nix, OS X or Windows
To install the Log Courier repository, download the corresponding `.repo`
configuration file below, and place it in `/etc/yum.repos.d`. Log Courier may
then be installed using `yum install log-courier`.

* **CentOS/RedHat 6.x**: [driskell-log-courier-epel-6.repo](https://copr.fedoraproject.org/coprs/driskell/log-courier/repo/epel-6/driskell-log-courier-epel-6.repo)
* **CentOS/RedHat 7.x**:
[driskell-log-courier-epel-7.repo](https://copr.fedoraproject.org/coprs/driskell/log-courier/repo/epel-6/driskell-log-courier-epel-7.repo)

***NOTE:*** *The RPM packages versions of Log Courier are built using ZeroMQ 3.2
and therefore do not support the encrypted `zmq` transport. They do support the
unencrypted `plainzmq` transport.*

### DEB

A Debian/Ubuntu compatible **PPA** repository is under consideration. At the
moment, no such repository exists.

## Building From Source

Requirements:

1. Linux, Unix, OS X or Windows
1. The [golang](http://golang.org/doc/install) compiler tools (1.2-1.4)
1. [git](http://git-scm.com)
1. GNU make

***\*nix:*** *Most requirements can usually be installed by your favourite package
manager.*
***Linux/Unix:*** *Most requirements can usually be installed by your favourite
package manager.*
***OS X:*** *Git and GNU make are provided automatically by XCode.*
***Windows:*** *GNU make for Windows can be found
[here](http://gnuwin32.sourceforge.net/packages/make.htm).*

### Building

To build, simply run `make` as follows.
To build the binaries, simply run `make` as follows.

git clone https://github.com/driskell/log-courier
cd log-courier
make

The log-courier program can then be found in the 'bin' folder.
The log-courier program can then be found in the 'bin' folder. This can be
manually installed anywhere on your system. Startup scripts for various
platforms can be found in the [contrib/initscripts](contrib/initscripts) folder.

*Note: If you receive errors whilst running `make`, try `gmake` instead.*

### Logstash Integration
## Logstash Integration

Log Courier does not utilise the lumberjack Logstash plugin and instead uses its
own custom plugin. This allows significant enhancements to the integration far
Expand All @@ -87,14 +142,29 @@ Install using the Logstash 1.5+ Plugin manager.
Detailed instructions, including integration with Logstash 1.4.x, can be found
on the [Logstash Integration](docs/LogstashIntegration.md) page.

### Building with ZMQ support
## Generating Certificates and Keys

To use the 'plainzmq' and 'zmq' transports, you will need to install
[ZeroMQ](http://zeromq.org/intro:get-the-software) (>=3.2 for cleartext
'plainzmq', >=4.0 for encrypted 'zmq').
Log Courier provides two commands to help generate SSL certificates and Curve
keys, `lc-tlscert` and `lc-curvekey` respectively. Both are bundled with the
packages provided by the public repositories.

***\*nix:*** *ZeroMQ >=3.2 is usually available via the package manager. ZeroMQ >=4.0
may need to be built and installed manually.*
When building from source, running `make selfsigned` will automatically build
and run the `lc-tlscert` utility that can quickly and easily generate a
self-signed certificate for the TLS shipping transport.

Likewise, running `make curvekey` will automatically build and run the
`lc-curvekey` utility that can quickly and easily generate CurveZMQ key pairs
for the CurveZMQ shipping transport. This tool is only available when Log
Courier is built with ZeroMQ >=4.0.

## ZeroMQ support

To use the 'plainzmq' or 'zmq' transports, you will need to install
[ZeroMQ](http://zeromq.org/intro:get-the-software) (>=3.2 for 'plainzmq', >=4.0
for 'zmq' which supports encryption).

***Linux\Unix:*** *ZeroMQ >=3.2 is usually available via the package manager.
ZeroMQ >=4.0 may need to be built and installed manually.*
***OS X:*** *ZeroMQ can be installed via [Homebrew](http://brew.sh).*
***Windows:*** *ZeroMQ will need to be built and installed manually.*

Expand All @@ -113,19 +183,6 @@ the Log Courier hosts are of the same major version. A Log Courier host that has
ZeroMQ 4.0.5 will not work with a Logstash host using ZeroMQ 3.2.4 (but will
work with a Logstash host using ZeroMQ 4.0.4.)**

### Generating Certificates and Keys

Running `make selfsigned` will automatically build and run the `lc-tlscert`
utility that can quickly and easily generate a self-signed certificate for the
TLS shipping transport.

Likewise, running `make curvekey` will automatically build and run the
`lc-curvekey` utility that can quickly and easily generate CurveZMQ key pairs
for the CurveZMQ shipping transport. This tool is only available when Log
Courier is built with ZeroMQ >=4.0.

Both tools also generate the required configuration file snippets.

## Documentation

* [Administration Utility](docs/AdministrationUtility.md)
Expand Down
29 changes: 16 additions & 13 deletions build/fix_version
Original file line number Diff line number Diff line change
@@ -1,32 +1,35 @@
#!/bin/bash

# If this is not a git repository, use the existing version
if [ ! -d '.git' ]; then
exit
fi

if [ -n "$1" ]; then
# When specified on the command line, it's always short, and means we're preparing a release
VERSION="$1"
VERSION_SHORT="$VERSION"
elif [ ! -d '.git' ]; then
# Not a git repository, so use the existing version_short.txt
VERSION="$(cat version_short.txt)"
VERSION_SHORT="$VERSION"
else
# Describe version from Git, and ensure the only "-xxx" is the git revision
# This ensures that gem builds only add one ".pre" tag automatically
VERSION="$(git describe | sed 's/-\([0-9][0-9]*\)-\([0-9a-z][0-9a-z]*\)$/.\1.\2/g')"
VERSION="${VERSION#v}"
VERSION_SHORT=$(git describe --abbrev=0)
VERSION_SHORT="${VERSION_SHORT#v}"
fi

# Patch version.go
sed "s/\\(const *Log_Courier_Version *string *= *\"\\)[^\"]*\\(\"\\)/\\1${VERSION}\\2/g" src/lc-lib/core/version.go > src/lc-lib/core/version.go.tmp
\mv -f src/lc-lib/core/version.go.tmp src/lc-lib/core/version.go
sed "s/<VERSION>/${VERSION}/g" src/lc-lib/core/version.go.tmpl > src/lc-lib/core/version.go

# Patch the gemspecs
for GEM in log-courier logstash-input-log-courier logstash-output-log-courier; do
sed "s/\\(gem.version *= *'\\)[^']*\\('\\)/\\1${VERSION}\\2/g" ${GEM}.gemspec > ${GEM}.gemspec.tmp
\mv -f ${GEM}.gemspec.tmp ${GEM}.gemspec
[ ${GEM#logstash-} != $GEM ] && {
sed "s/\\(gem.add_runtime_dependency *'log-courier' *, *'= *\\)[^']*\\('\\)/\\1${VERSION}\\2/g" ${GEM}.gemspec > ${GEM}.gemspec.tmp
\mv -f ${GEM}.gemspec.tmp ${GEM}.gemspec
}
sed "s/<VERSION>/${VERSION}/g" ${GEM}.gemspec.tmpl > ${GEM}.gemspec
done

# Store the full version in version.txt for other scripts to use, such as push_gems
echo "${VERSION}" > version.txt

# Store the nearest tag in version_short.txt - this is the only file stored in the repo
# This file is used as the version if we download Log Courier as a non-git package
echo "${VERSION_SHORT}" > version_short.txt

echo "Set Log Courier Version ${VERSION}"
2 changes: 1 addition & 1 deletion build/push_gems
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

for GEM in *-$(cat version.txt).gem; do
for GEM in *-$(cat version_short.txt).gem; do
echo "- ${GEM}"
gem push $GEM
done
Expand Down
5 changes: 5 additions & 0 deletions build/setup_root
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#!/bin/bash

# Allow the source code to refer to github.com/driskell/log-courier paths
mkdir -p src/github.com/driskell
ln -nsf ../../.. src/github.com/driskell/log-courier
Loading

0 comments on commit e7423cf

Please sign in to comment.