-
Notifications
You must be signed in to change notification settings - Fork 0
Software & Deployment Architecture
The overall architecture is:
- Individual models in their own Go packages, abstracted enough to be Lambda or binary
- Individual models and logic as a Lambda, supported by a single endpoint in API Gateway
- A single DynamoDB Table backing each model
- Serverless for deploying all functions into AWS Lambda
For the software's architecture, we're going to define a few key patterns to follow. This isn't to put in place hard and fast rules, but instead, simply ensure we have a path to follow.
When writing a Lambda for an API Gateway endpoint, which essentially represents a model in the application, we will create a new sub-package under the SWL root path. This means each endpoint is an individual package and can be:
- Tested in isolation
- Can be a Lambda function deployed via Serverless
- Can be a standalone application
- Can be used in a bigger, stand alone project as a module - i.e. redeployed outside of Lambda as a monolith
I believe this structure will give us flexibility and make deployments via Serverless very easy.
What we need here to ensure each sub-package, as defined above, can actually be deployed as a Lambda or a binary. This is somewhat a repeat of the above, but it's important to be clear because it requires smart choices during the code base's inception.
In short, we simply need to ensure that we're writing a RESTful API (as close to RESTful as we want or need to get to, anyway) that can be interacted with via a Lambda Handler OR via a Go net/http server. This means keeping each endpoint's core business logic fully separate from the code handling the incoming connection.
Let's consider an example. If we have an endpoint that takes in a name and stores it in a database, we want to make sure the logic for storing the name is kept in a separate function to the logic that a) handles the Lambda invocation and b) the logic that handles a direct HTTP(S) connection from the public Internet, or even c) a connection from some other protocol, like something we make our selves using TCP, UDP, or whatever. Essentially, whatever handles the inbound connection needs to very quickly pass the work onto the business logic and then only return the result.
Selecting and defining the libraries we use in our code isn't setting anything in stone, but it's just making sure we know what direction we want to go in. Let's cover what I believe are the correct libraries to be using, and why.
This library is one of the best net/http replacements available. It's very simple and easy to use, and it's extremely fast. This is a perfect choice for handling our connections coming over a direct HTTP(S) connection instead of via a Lambda handler.
When it comes to configuration from files, the environment, remote APIs, and other sources, nothing beats Viper. It's an incredibly powerful library that makes handling configuration for your application a complete breeze. It really has no competition at all.
As we're going to be using DynamoDB, we'll need to be able to mock it for testing purposes. This creating speeds up the ability to test our software changes without having to incur charges or suffer latency issues as we wait for requests to be completed by the real AWS DynamoDB.
There may be times when we abstract away functionality and require an interface. When this happens, it's likely we will need and want to create a mock implementation of the interface: step in moq. It's an excellent library that will read an interface's definition and create a concrete implementation for use in tests.
With gotests, we can very quickly generate table driven unit tests for all of our functions and even receivers on structs. Combined with moq, it leaves virtually no excuse for not having tests in place as it outputs all the boiler plate unit testing code, only leaving you the task of refining the tests and adding in test cases.
This one should be somewhat obvious, really. We'll likely need to use this library for various interactions with AWS services such as DynamoDB, S3, CloudWatch Logs, etc. It will become quickly apparent whether or not we need this library.
For managing our libraries in our Go based code, we shall use the dep dependency management utility. It's extremely good, and is actually becoming, or might have become, the official dependency management tool for Go.
We need to write a generic, common handler for each package so that the inbound request is being handled in a highly consistent manner and then quickly being handed off to some internal function. This sounds somewhat obvious or maybe it sounds restrictive, but what we're trying to do here is simply keep things simple and consistent among all the handlers so there are no surprises when coming back later on to maintain or update the code.
Because we're handling inbound traffic from the public, we need to define an idea around what the URLs are going to look like. It doesn't have to be anything concrete now, just a guideline to work from.
When providing URLs for the client to work with, we need to make them somewhat obvious and simple to understand. Let's begin by defining the interactions people will have with our API:
- Create, update, and delete an account
- Create, update, and delete wish lists
- Create, update, and delete items for wish lists
- Add and remove friends from their account
- Add and remove friends from wish lists on their account (invitations)
- Login and logout from their account
Based on these simple CRUD (Create, Read, Update, Delete) operations, we can create endpoints along these lines:
/account//account/create/account/update/account/delete/account/find
/wishlist//wishlist/create/wishlist/update/wishlist/delete/wishlist/invite/wishlist/accessible
/item//item/create/item/update/item/delete
/friend//friend/add/friend/remove
/invitation//invitation/add/invitation/remove
/login/logout
We're going to need models for each of the endpoints we've listed above. These models represent state, and that state is stored in DynamoDB Tables. We will add, update, and remove state from the Tables as and when the user interacts with the service.
The design we're taking here is to define each model as a struct{} which complies with a particular interface to ensure certain methods are available on each model, such as being able to Add(), Remove(), and Update() model entries from its respective Table.
We will need an interface for our database access. I have two goals in mind with regards to database access:
- I want to be able to push to DynamoDB today
- I want to be able to push to any other database tomorrow
It's possible we'll expand the project to include a time series database, or some kind of interaction with a remote API. For us to be able to do this, we need to abstract the API each model uses for databases access. We can do this using an interface.
There's also the topic of mocking for testing purposes. We don't want to run a local DynamoDB instance (such a thing exists) just to run tests. That's too heavy. Instead we can mock the database access interface for each model and uses mock data, too.
This is going to be a bigger section than above, possibly the biggest in the documentation, but it's an important one and will evolve a lot over time.
Keep in mind that at the time of writing, these models have been kept simple for v1. It's highly likely these models will change a lot over the course of development as feedback comes in from the public and other developers.
Note: Login and Profile seem like the same thing, but it's considered best practice to keep them separated. This enables the user to change their password, username, email address, etc, all independently of their profile. It means they can also have two or more logins per profile, such as via a username/password combination or an API key. You can do other funky things with this separation too, like handing over an account to someone else, like a transfer, by simply transferring your Profile under their Login so that the user has two profiles to play with. Not all of these features will be implemented, they're just examples.
Each model will have a database interface associated with it, also. See above.
Login
type Login struct {
ID string
OwnerID string
Username string
PasswordHash string
Enabled bool
Verified bool
}
type DBI interface {
GetByID(string) (Login, error)
GetByOwnerID(string) (Login, error)
GetByUsername(string) (Login, error)
Add(Login) error
Delete(string) error
Update(Login) error
}Profile
type Profile struct {
ID string
LoginIDs []string
RealName string
Email string
DOB time.Time
Friends []Friend
WishLists []WishList
}
type DBI interface {
GetByID(string) (Profile, error)
GetByRealName(string) (Profile, error)
GetByEmail(string) (Profile, error)
Add(Profile) error
Delete(string) error
Update(Profile) error
}WishList
type WishList struct {
ID string
OwnerID string
Title string
Description string
Active bool
Private bool
Items []WishListItem
Friends []Friend
}
type DBI interface {
GetByID(string) (WishList, error)
GetByOwnerID(string) (WishList, error)
Add(WishList) error
Delete(WishList) error
Update(WishList) error
GetItems(WishList) ([]WishListItem, error)
GetFriends(WishList) ([]Friend, error)
InviteFriends(string, []Friend) error
RemoveFriends(string, []Friend) error
}Item
type WishListItem struct {
ID string
OwnerID string
ListID string
Title string
Description string
Cost float32
Rating uint
Bought bool
BeingBought bool
Active bool
}
type DBI interface {
GetByID(string) (WishListItem, error)
GetByOwnerID(string) (WishListItem, error)
GetByListID(string) (WishListItem, error)
Add(WishListItem) error
Delete(string) error
Update(WishListItem) error
}Friend
type FriendStatus uint
const (
noresponse = FriendStatus(0)
accepted = FriendStatus(1)
rejected = FriendStatus(2)
)
type Friend struct {
ID string
OwnerID string
ProfileID string
Status FriendStatus
}
type DBI interface {
GetByID(string) (Friend, error)
GetByOwnerID(string) (Friend, error)
GetByProfileID(string) (Friend, error)
Add(Friend) error
Delete(string) error
Update(Friend) error
}The concept of deploying code is a simple one: get your product out there and live for the world to see and use. We're going to outline some basics around what we're deploying and how.
Our application is going to use AWS Lambda and API Gateway for handling the compute resources and API, respectively. Lambda is going to house our functions and each function will handle one HTTP(S) endpoint (which we defined above.) We will use API Gateway to route traffic from the public Internet to individual functions based on the URL requested. This is pretty simple to understand, but here's an ASCII diagram demonstrating the traffic flow:
Client -> Internet -> API Gateway -> Lambda (-> Maybe DynamoDB, maybe some cache)
To get this in place, we're going to use Serverless.
The Serverless Framework is an excellent piece of kit. It enables us to define what our application looks like based on routes and functions, among other things, and it produces the CloudFormation required to get it setup and running. It doesn't just dump CloudFormation in our lap, however. It actually applies the CloudFormation for us, so with one command we can have a whole application up and running.
In this document we won't discuss or look at specifics. Instead we will do that in another document in the wiki.
What we're doing here is defining a single serverless.yml file for deploying the entire stack. This will be helped along nicely with our code structure as discussed earlier. Each function will be compiled and placed into a sub-package, meaning all Serverless has to do pair up an API Gateway route/endpoint with a Lambda function, pass through the arguments, and we're done.
With a single command, sls deploy, we'll have our application deployed.
And finally, we will make sure that the serverless.yml file supports us passing in an environment name, allowing each deployment to be unique and isolated. This has proven useful to me in the past when wanting to stand up multiple instances of an application for testing or for developers to use.