"ToDo API" Microservice Example

Introduction

Welcome! 👋

This is an educational repository that includes a microservice written in Go. It is used as the principal example of my video series: Building Microservices in Go.

This repository is not a template nor a framework, it's a collection of patterns and guidelines I've successfully used to deliver enterprise microservices when using Go, and just like with everything in Software Development some trade-offs were made.

My end goal with this project is to help you learn another way to structure your Go project with 3 final goals:

1. It is enterprise, meant to last for years,
2. It allows a team to collaborate efficiently with little friction, and
3. It is as idiomatic as possible.

Join the fun at https://youtube.com/MarioCarrion.

Domain Driven Design

This project uses a lot of the ideas introduced by Eric Evans in his book Domain Driven Design, I do encourage reading that book but before I think reading Domain-Driven Design Distilled makes more sense, also there's a free to download DDD Reference available as well.

On YouTube I created a playlist that includes some of my favorite talks and webinars, feel free to explore that as well.

Project Structure

Talking specifically about microservices only, the structure I like to recommend is the following, everything using < and > depends on the domain being implemented and the bounded context being defined.

• build/: defines the code used for creating infrastructure as well as docker containers.
  • <cloud-providers>/: define concrete cloud provider.
  • <executableN>/: contains a Dockerfile used for building the binary.
• cmd/
  • <primary-server>/: uses primary database.
  • <replica-server>/: uses readonly databases.
  • <binaryN>/
• db/
  • migrations/: contains database migrations.
  • seeds/: contains file meant to populate basic database values.
• internal/: defines the core domain.
  • <datastoreN>/: a concrete repository used by the domain, for example postgresql
  • http/: defines HTTP Handlers.
  • service/: orchestrates use cases and manages transactions.
• pkg/ public API meant to be imported by other Go package.

There are cases where requiring a new bounded context is needed, in those cases the recommendation would be to define a package like internal/<bounded-context> that then should follow the same structure, for example:

• internal/<bounded-context>/
  • internal/<bounded-context>/<datastoreN>
  • internal/<bounded-context>/http
  • internal/<bounded-context>/service

Tools

Please refer to the documentation in internal/tools/.

Features

Icons meaning:

• means a link to Youtube video.
• means a link to Blog post.

In no particular order:

• Project Layout
• Dependency Injection
• Secure Configuration
  • Using Hashicorp Vault
  • Using AWS SSM
• Infrastructure as code
• Metrics, Traces and Logging using OpenTelemetry
• Caching
  • Memcached
  • Redis
• Persistent Storage
  • Repository Pattern
  • Database migrations
  • MySQL
  • PostgreSQL
    • jmoiron/sqlx, jackc/pgx and database/sql
    • go-gorm/gorm and volatiletech/sqlboiler
    • Masterminds/squirrel and kyleconroy/sqlc
• REST APIs
  • HTTP Handlers
  • Custom JSON Types
  • Versioning
  • Error Handling
  • OpenAPI 3 and Swagger-UI
  • Authorization
• Events and Messaging
  • Apache Kafka
  • RabbitMQ
  • Redis
• Testing
  • Type-safe mocks with maxbrunsfeld/counterfeiter
  • Equality with google/go-cmp
  • Integration tests for Datastores with ory/dockertest
  • REST APIs
• Containerization using Docker
• Graceful Shutdown
• Search Engine using ElasticSearch
• Documentation
  • C4 Model
• Cloud Design Patterns
  • Reliability
    • Circuit Breaker
• Tools as Dependencies
• Whatever else I forgot to include

More ideas

• 2016: Peter Bourgon's: Repository structure
• 2016: Ben Johnson's: Standard Package Layout
• 2017: William Kennedy's: Design Philosophy On Packaging
• 2017: Jaana Dogan's: Style guideline for Go packages
• 2018: Kat Zien - How Do You Structure Your Go Apps

Docker Containers

Please notice in order to run this project locally you need to run a few programs in advance, if you use Docker please refer to the concrete instructions in docs/ for more details.

There's also a docker-compose.yml, covered in Building Microservices In Go: Containerization with Docker, however like I mentioned in the video you have to execute docker-compose in multiple steps.

Notice that because of the way RabbitMQ and Kafka are being used they are sort of competing with each other, so at the moment we either have to enable Kafka and disable RabbitMQ or the other way around in both the code and the docker-compose.yml file, in either case there are Dockerfiles and services defined that cover building and running them.

• Run docker-compose up, here both rest-server and elasticsearch-indexer services will fail because the postgres, rabbitmq, elasticsearch and kafka services take too long to start.
  • If you're planning to use RabbitMQ, run docker-compose up rest-server elasticsearch-indexer-rabbitmq.
  • If you're planning to use Kafka, run docker-compose up rest-server elasticsearch-indexer-kafka.
  • If you're planning to use Redis, run docker-compose up rest-server elasticsearch-indexer-redis.
• For building the service images you can use:
  • rest-server image: docker-compose build rest-server.
  • elasticsearch-indexer-rabbitmq image: docker-compose build elasticsearch-indexer-rabbitmq.
  • elasticsearch-indexer-kafka image: docker-compose build elasticsearch-indexer-kafka.
  • elasticsearch-indexer-redis image: docker-compose build elasticsearch-indexer-redis.
• Run docker-compose run rest-server migrate -path /api/migrations/ -database postgres://user:password@postgres:5432/dbname?sslmode=disable up to finally have everything working correctly.

Diagrams

To start a local HTTP server that serves a graphical editor:

mdl serve github.com/MarioCarrion/todo-api/internal/doc -dir docs/diagrams/

To generate JSON artifact for uploading to structurizr:

stz gen github.com/MarioCarrion/todo-api/internal/doc