gologaggregator

High-performance, multi-protocol log aggregator built in Go with concurrent ingestion and processing.

---

gologaggregator is a high-performance log aggregation service that supports multiple ingestion protocols simultaneously. Designed for scalability, it leverages Go's concurrency primitives to ingest, process, and index logs with minimal latency and high resilience through disk persistence (WAL).

Demonstration

Ingestion via HTTP (JSON)

curl -X POST http://localhost:8080/logs \
  -H "Content-Type: application/json" \
  -d '{"content": "INFO: System initialized successfully"}'

Ingestion via TCP (Plain Text)

echo "ERROR: Database connection failure" | nc localhost 5000

Searching Logs

curl "http://localhost:8080/search?level=ERROR"

Tech Stack

Technology	Role
Go 1.25	Core language and concurrent execution
Net/HTTP	Protocol implementation and REST API
Docker	Containerization and load simulation
Sync/Mutex	Thread-safety guarantee for in-memory index

Prerequisites

• Go >= 1.25 (defined in go.mod)
• Docker & Docker Compose (optional)
• Network tools (curl, nc/telnet)

Installation and Usage

As a Binary

go install github.com/ESousa97/gologaggregator/cmd/aggregator@latest

From Source

git clone https://github.com/ESousa97/gologaggregator.git
cd gologaggregator
# Optional: cp .env.example .env
make build
make run

Makefile Targets

Target	Description
make build	Compiles the project binary to bin/
make run	Runs the aggregator directly via go run
make test	Runs all unit tests in the project
make lint	Runs the linter (golangci-lint or go vet)
make docker-up	Starts the full environment via docker-compose
make docker-down	Stops the docker-compose services
make clean	Removes build artifacts and temporary files
make spell	Runs spell checker (requires cspell)

Architecture

The project adopts a modular architecture focused on separation of concerns:

• internal/tcp & internal/http: Decoupled ingestion drivers.
• internal/pipeline: Core of the system, manages Worker Pools and log buffering.
• internal/parser: Logic for level extraction and message structuring.
• internal/store: Thread-safe in-memory index (Circular Buffer).
• internal/persistence: Disk writing layer with rotation support.

API Reference

Detailed documentation of the API and internal packages can be found at pkg.go.dev.

Configuration

The application uses environment variables or defaults.

Variable	Description	Type	Default
SERVER_HOST	Host for TCP/HTTP servers	string	0.0.0.0
TCP_PORT	Port for raw TCP ingestion	int	5000
HTTP_PORT	Port for REST API and HTTP ingestion	int	8080

Roadmap

• Multi-Protocol Ingestion (TCP/HTTP)
• Concurrent processing pipeline with Worker Pool
• In-memory indexing with search and filtering
• Structured Query API
• Disk persistence and log rotation

Contributing

Contributions are welcome! See the full guide at CONTRIBUTING.md.

License

Distributed under the MIT license. See LICENSE for more details.

Author

Enoque Sousa

⬆ Back to top

Made with ❤️ by Enoque Sousa

Project Status: Archived — Study Project