Skip to content

Commit

Permalink
Stage hooks (#23)
Browse files Browse the repository at this point in the history 
* Improve README.md

* Split pipeline and implement hooks
  • Loading branch information
@jwillp
jwillp authored Sep 7, 2024
1 parent 5e39de9 commit 1304127
Show file tree
Hide file tree
Showing 36 changed files with 2,305 additions and 1,394 deletions.
111 changes: 102 additions & 9 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,110 @@

[![Go](https://github.com/Morebec/specter/actions/workflows/go.yml/badge.svg)](https://github.com/Morebec/specter/actions/workflows/go.yml)

Specter is a development toolkit in Go that allows you to develop configuration file processors based on
HashiCorp Configuration Language (HCL). With Specter, you can define your own Domain-Specific Language (DSL)
using HCL and create a processing pipeline to validate, lint, resolve dependencies, and generate code or artifact
files from these DSL configuration files.
Specter is a Go library designed to help developers easily build declarative DSLs (Domain-Specific Languages) and
process them through an extensible pipeline.

## Features
It is currently used at [Morébec](https://morebec.com) for generating microservice APIs, code and documentation,
managing configurations, automating deployments, and so many other fun things.

- Develop your own DSL using HCL
- Validate and lint configuration files
- Resolve dependencies between configuration files
- Generate code or artifact files from configuration files
Specter provides a simple yet powerful framework to simplify these workflows.

The library also comes with many batteries included for common tasks such as dependency resolution
and linting, HCL configuration loading and more.

## Key Use Cases

At [Morébec](https://morebec.com) Specter is primarily used to create high-level, syntax-consistent DSLs for tools
like OpenAPI, Docker/Docker Compose, and Terraform.

Here are some of the key use cases Specter powers for us internally:

- **Code Generation:** We generate entire code bases in PHP and Go leveraging DDD/CQRS/ES in a low-code manner to focus on business logic and
reduce plumbing work.
- **Enforce Coding Standards**: We ensure consistency and improve development speed by automating code quality checks and
standardization.
- **Configuration Management:** We use it to manage environment-specific configuration files, such as Docker or
Kubernetes manifests, using declarative Units.
- **CI/CD Automation:** Automate the generation of CI/CD pipeline definitions (Jenkins, GitHub Actions, etc.)
by processing high-level declarative Units.
- **Infrastructure as Code:** Describe infrastructure components declaratively and generate Terraform,
scripts, or other IAC artifacts.


## How Specter Works
Specter is based around a simple yet powerful *pipeline* architecture. The core of the pipeline is designed to process
*Units* — which are declarative components that represent different aspects or concepts — and produce various types
of outputs based on them called *artifacts*.

In short, specter loads Units which it processes before outputting corresponding artifacts.

For example, in the case of our Go code generator, we first define Microservices with their Commands, Events
and Queries in specification files that are then processed by Specter and transformed into their
corresponding Go implementation along with a changelog, markdown documentation and OpenAPI specification.

In this example, the Microservice/Command/Event/Query definition files are the "Units", while the
generated code, markdown documentation, changelog, and OpenAPI are the "artifacts".

Units are anything that needs transforming, and artifacts are anything these units can be transformed into.

To illustrate, here's an example of a Unit File that could describe a docker container to be deployed on a
given host using an HCL syntax:

```
service "web" {
image = "our-app:latest"
ports = ["8080:80"]
volumes = [
{
type = "bind"
source = "./html"
target = "/usr/share/nginx/html"
}
]
deploymentHost = "primary-us-east-dev"
}
```

### Pipeline Stages
The pipeline consists of several stages, each responsible for a specific task in the workflow.
Here's an overview of the stages and the concepts they introduce:

### 1. Source Loading
The very first step is to acquire these units. Depending on the use cases these units could come from files, HTTP resources,
or even Database rows. These different locations are known in Specter as Unit Sources.

As such, the Source Loading stage corresponds to loading these sources so that they can be acquired/fetched
and read.

- Inputs: Source locations
- Outputs: (Loaded) Sources

### 2. Unit Loading
Units are read and materialized into in-memory data structures. This stage converts raw source data into
usable Units that can be processed according to your specific needs.

- Inputs: Sources
- Outputs: (Loaded) Units

### 3. Unit Processing
Once the Units are loaded, Specter applies processors which are the core services responsible for generating artifacts
based on these units. These processors can do things like validate the Units, resolve dependencies, or convert them
into different representations. You can easily add custom processors to extend Specter's behavior.

The artifacts are in-memory representations of the actual desired outputs. For instance, the FileArtifact represents
a file to be outputted.

- Inputs: Sources
- Outputs: Artifacts

### 4. Artifact Processing
The final stage of the pipeline processes artifacts that were generated during the previous step.
The processing of these artifacts can greatly vary based on the types of artifacts at play.
An artifact could be anything from a file, an API call, to a database insertion or update query,
to a command or program to be executed.

- Inputs: Artifacts
- Outputs: Final outputs (files, API calls, etc.)

## Getting Started

Expand Down
209 changes: 15 additions & 194 deletions pkg/specter/artifactproc.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,17 @@ package specter

import (
"context"
"encoding/json"
"github.com/morebec/go-errors/errors"
"io/fs"
"os"
"slices"
"sync"
"time"
)

// ArtifactProcessor are services responsible for processing artifacts of UnitProcessors.
type ArtifactProcessor interface {
// Process performs the processing of artifacts generated by UnitProcessor.
Process(ctx ArtifactProcessingContext) error

// Name returns the name of this processor.
Name() string
}

// ArtifactRegistry provides an interface for managing a registry of artifacts. This
// registry tracks artifacts generated during processing runs, enabling clean-up
// in subsequent runs to avoid residual artifacts and maintain a clean slate.
Expand Down Expand Up @@ -97,199 +99,18 @@ type ArtifactProcessingContext struct {
context.Context
Units UnitGroup
Artifacts []Artifact
Logger Logger

ArtifactRegistry ProcessorArtifactRegistry
processorName string
}

var _ ArtifactRegistry = (*InMemoryArtifactRegistry)(nil)

// InMemoryArtifactRegistry maintains a registry in memory.
// It can be useful for tests.
type InMemoryArtifactRegistry struct {
EntriesMap map[string][]ArtifactRegistryEntry
mu sync.RWMutex // Mutex to protect concurrent access
}

func (r *InMemoryArtifactRegistry) Add(processorName string, e ArtifactRegistryEntry) error {
if processorName == "" {
return errors.NewWithMessage(errors.InternalErrorCode, "processor name is required")
}
if e.ArtifactID == "" {
return errors.NewWithMessage(errors.InternalErrorCode, "artifact id is required")
}

r.mu.Lock()
defer r.mu.Unlock()

if r.EntriesMap == nil {
r.EntriesMap = map[string][]ArtifactRegistryEntry{}
}

if _, ok := r.EntriesMap[processorName]; !ok {
r.EntriesMap[processorName] = make([]ArtifactRegistryEntry, 0)
}

for i, entry := range r.EntriesMap[processorName] {
if entry.ArtifactID == e.ArtifactID {
r.EntriesMap[processorName] = slices.Delete(r.EntriesMap[processorName], i, i+1)
}
}

r.EntriesMap[processorName] = append(r.EntriesMap[processorName], e)

return nil
}

func (r *InMemoryArtifactRegistry) Remove(processorName string, artifactID ArtifactID) error {
r.mu.Lock()
defer r.mu.Unlock()

if processorName == "" {
return errors.NewWithMessage(errors.InternalErrorCode, "processor name is required")
}
if artifactID == "" {
return errors.NewWithMessage(errors.InternalErrorCode, "artifact id is required")
}

if _, ok := r.EntriesMap[processorName]; !ok {
return nil
}

var artifacts []ArtifactRegistryEntry
for _, entry := range r.EntriesMap[processorName] {
if entry.ArtifactID != artifactID {
artifacts = append(artifacts, entry)
}
}

r.EntriesMap[processorName] = artifacts

return nil
}

func (r *InMemoryArtifactRegistry) FindByID(processorName string, artifactID ArtifactID) (entry ArtifactRegistryEntry, found bool, err error) {
all, _ := r.FindAll(processorName)

for _, e := range all {
if e.ArtifactID == artifactID {
return e, true, nil
}
}

return ArtifactRegistryEntry{}, false, nil
}

func (r *InMemoryArtifactRegistry) FindAll(processorName string) ([]ArtifactRegistryEntry, error) {
if r.EntriesMap == nil {
return nil, nil
}

values, ok := r.EntriesMap[processorName]
if !ok {
return nil, nil
}

return values, nil
}

func (r *InMemoryArtifactRegistry) Load() error { return nil }

func (r *InMemoryArtifactRegistry) Save() error { return nil }

const DefaultJSONArtifactRegistryFileName = ".specter.json"

type JSONArtifactRegistryRepresentation struct {
GeneratedAt time.Time `json:"generatedAt"`
EntriesMap map[string][]JSONArtifactRegistryEntry `json:"entries"`
type ArtifactProcessorFunc struct {
name string
processFunc func(ctx ArtifactProcessingContext) error
}

type JSONArtifactRegistryEntry struct {
ArtifactID string `json:"artifactId"`
Metadata map[string]any `json:"metadata"`
func (a ArtifactProcessorFunc) Process(ctx ArtifactProcessingContext) error {
return a.processFunc(ctx)
}

var _ ArtifactRegistry = (*JSONArtifactRegistry)(nil)

// JSONArtifactRegistry implementation of a ArtifactRegistry that is saved as a JSON file.
type JSONArtifactRegistry struct {
*InMemoryArtifactRegistry
FileSystem FileSystem
FilePath string
TimeProvider TimeProvider

mu sync.RWMutex // Mutex to protect concurrent access
}

func (r *JSONArtifactRegistry) Load() error {
r.mu.Lock()
defer r.mu.Unlock()

bytes, err := r.FileSystem.ReadFile(r.FilePath)
if err != nil {
if os.IsNotExist(err) {
return nil
}
return errors.WrapWithMessage(err, errors.InternalErrorCode, "failed loading artifact file registry")
}

// empty file is okay
if len(bytes) == 0 {
return nil
}

repr := &JSONArtifactRegistryRepresentation{}

if err := json.Unmarshal(bytes, repr); err != nil {
return errors.WrapWithMessage(err, errors.InternalErrorCode, "failed loading artifact file registry")
}

for processorName, entries := range repr.EntriesMap {
for _, entry := range entries {
if err := r.InMemoryArtifactRegistry.Add(processorName, ArtifactRegistryEntry{
ArtifactID: ArtifactID(entry.ArtifactID),
Metadata: entry.Metadata,
}); err != nil {
return err
}
}
}

return nil
}

func (r *JSONArtifactRegistry) Save() error {
r.mu.RLock()
defer r.mu.RUnlock()

repr := JSONArtifactRegistryRepresentation{
GeneratedAt: r.TimeProvider(),
EntriesMap: make(map[string][]JSONArtifactRegistryEntry, len(r.InMemoryArtifactRegistry.EntriesMap)),
}

// Add entries to representation
for processorName, entries := range r.InMemoryArtifactRegistry.EntriesMap {
repr.EntriesMap[processorName] = nil
for _, entry := range entries {
repr.EntriesMap[processorName] = append(repr.EntriesMap[processorName], JSONArtifactRegistryEntry{
ArtifactID: string(entry.ArtifactID),
Metadata: entry.Metadata,
})
}
}

// Set generation date
repr.GeneratedAt = r.TimeProvider()

// Generate a JSON file containing all artifact files for clean up later on
js, err := json.MarshalIndent(repr, "", " ")
if err != nil {
return errors.Wrap(err, "failed generating artifact file registry")
}
if err := r.FileSystem.WriteFile(r.FilePath, js, fs.ModePerm); err != nil {
return errors.Wrap(err, "failed generating artifact file registry")
}

return nil
}
func (a ArtifactProcessorFunc) Name() string { return a.name }
Loading

0 comments on commit 1304127

Please sign in to comment.