diff --git a/README.md b/README.md index da552e1..518b52e 100644 --- a/README.md +++ b/README.md @@ -8,30 +8,70 @@ ## Table of Contents -- [DriveDeposits System Design: Integrated Microservices Architecture](#drivedeposits-system-design-integrated-microservices-architecture) +- [DriveDeposits System Design and Microservices Architecture: Watch Ingestion and Query Workflows](#drivedeposits-system-design-and-microservices-architecture-watch-ingestion-and-query-workflows) + - [System Design](#system-design) + - [Workflow Videos](#workflow-videos) + - [Ingestion Workflow Videos](#ingestion-workflow-videos) + - [Valid Portfolio Request](#valid-portfolio-request) + - [Portfolio request and response](#portfolio-request-and-response) + - [Query Workflow Videos](#query-workflow-videos) + - [Query API Retrieves a list of portfolios based on the delta growth criteria](#query-api-retrieves-a-list-of-portfolios-based-on-the-delta-growth-criteria) + - [Query API Retrieves a list of banks for a portfolio based on the delta growth criteria](#query-api-retrieves-a-list-of-banks-for-a-portfolio-based-on-the-delta-growth-criteria) + - [Query API Retrieves a list of deposits for a portfolio based on the delta growth criteria](#query-api-retrieves-a-list-of-deposits-for-a-portfolio-based-on-the-delta-growth-criteria) + - [Query API Retrieves a list of deposits for a portfolio based on the maturity date criteria](#query-api-retrieves-a-list-of-deposits-for-a-portfolio-based-on-the-maturity-date-criteria) - [Domain Driven Terminology](#domain-driven-terminology) - [DriveDeposits: Architectural Pillars](#drivedeposits-architectural-pillars) - [Synchronous Microservices Components](#synchronous-microservices-components) - [Asynchronous Microservices Components](#asynchronous-microservices-components) - [Bridging Synchronous and Asynchronous Components In DriveDeposits Microservices](#bridging-synchronous-and-asynchronous-components-in-drivedeposits-microservices) - [Deployment of Microservices](#deployment-of-microservices) -- [Hybrid Integration Testing Tool](#hybrid-integration-testing-tool) + - [Serverless: AWS using SAM which uses CloudFormation under the hood](#serverless-aws-using-sam-which-uses-cloudformation-under-the-hood) + - [Server-based: Run the REST and gRPC servers Natively, With Docker Compose And Kubernetes!](#server-based-run-the-rest-and-grpc-servers-natively-with-docker-compose-and-kubernetes) + - [Test microservices integration](#test-microservices-integration) + - [Test AWS Lambda microservice directly](#test-aws-lambda-microservice-directly) + - [Data population](#data-population) + - [Querying with custom domain](#querying-with-custom-domain) - [Running Tests](#running-tests) - [Integration tests](#integration-tests) - [Unit and Integration tests](#unit-and-integration-tests) - [End-to-End tests](#end-to-end-tests) -- [Data population](#data-population) -- [Querying with custom domain](#querying-with-custom-domain) + - [Hybrid Integration Testing Tool](#hybrid-integration-testing-tool) + - [Efficient Command for Synchronous Calculation Flow and Asynchronous AWS Serverless Event Testing](#efficient-command-for-synchronous-calculation-flow-and-asynchronous-aws-serverless-event-testing) + - [Alias](#alias) - [Development Tool: cargo lambda](#development-tool-cargo-lambda) - [Development Tool: LocalStack](#development-tool-localstack) - [Clean And Build](#clean-and-build) - [Configurations for DriveDeposits](#configurations-for-drivedeposits) - [Member crates in workspace](#member-crates-in-workspace) -### DriveDeposits System Design: Integrated Microservices Architecture +### DriveDeposits System Design and Microservices Architecture: Watch Ingestion and Query Workflows + +#### System Design ![DriveDeposits Design](DriveDeposits.drawio.svg) +#### Workflow Videos + +#### Ingestion Workflow Videos + +##### Valid Portfolio Request + +##### [![Valid portfolio request](media/valid-portfolio-request.png)](https://vimeo.com/1042543651) + +##### Portfolio request and response + +##### [![Portfolio request and response](media/portfolio-request-and-response.png)](https://vimeo.com/1042543661) + +#### Query Workflow Videos + +#### [Query API Retrieves a list of portfolios based on the delta growth criteria](https://vimeo.com/1042543613) + +#### [Query API Retrieves a list of banks for a portfolio based on the delta growth criteria](https://vimeo.com/1042543640) + +#### [Query API Retrieves a list of deposits for a portfolio based on the delta growth criteria](https://vimeo.com/1042543632) + +#### [Query API Retrieves a list of deposits for a portfolio based on the maturity date criteria](https://vimeo.com/1042543620) + [Back to Table of Contents](#table-of-contents) ### Domain Driven Terminology: @@ -134,8 +174,9 @@ domain-specific language in both design and development: ``` -* **Sorting Capabilities with Top K Based on Delta Growth:** DriveDeposits allows sorting based on delta growth, - retrieving the top 'k' items (where 'k' is the number defined by the user in the query) in ascending or descending +* **Sorting Capabilities with Top K Based on Delta Growth:** DriveDeposits allows sorting based on delta growth by + portfolio, bank and deposits level and maturity date at deposits level retrieving the top 'k' items (where 'k' is the + number defined by the user in the query) in ascending or descending order. For example: ```curl @@ -257,7 +298,7 @@ microservices ecosystem. ### Deployment of Microservices -- **Serverless**: AWS using SAM which uses CloudFormation under the hood +#### Serverless: AWS using SAM which uses CloudFormation under the hood This project uses SAM (Serverless Application Model) for deploying the following AWS resources that support our microservices architecture: @@ -274,217 +315,184 @@ microservices application. This project also utilizes the [Justfile](https://github.com/casey/just) for managing project-level recipes and microservices deployment tasks. -#### Deploy all serverless microservices and AWS resources +- **Deploy all serverless microservices and AWS resources** `just deploy-drive-deposits-dynamodb-queries` This command calls dependent recipes to deploy the entire microservices ecosystem, including the event bus, event rules with lambda targets, and lambda functions for queries. -#### Delete all serverless microservices and AWS resources +- **Delete all serverless microservices and AWS resources** `just deployed-delete-drive-deposits-event-bus` This command calls dependent recipes to delete all deployed microservices and resources, including event rules, lambda targets, event bus, and query-related resources. -#### Deploy event bridge with event rules and target lambda microservices components +- **Deploy event bridge with event rules and target lambda microservices components** `just deploy-drive-deposits-event-rules` This command deploys the event-driven components of our microservices architecture, including EventBridge, EventBus, and related resources. -#### Deploy Lambda Function with API Gateway and DynamoDB Reader microservices components +- **Deploy Lambda Function with API Gateway and DynamoDB Reader microservices components** `just deploy-drive-deposits-dynamodb-queries-only` -- **Server-based**: Run the REST and gRPC servers Natively, With Docker Compose Or Kubernetes! - - **Natively** (without Docker) - - `just run-drive-deposits-grpc-server` - - `just run-drive-deposits-rest-grpc-gateway-server` - - - **Docker Compose** - Start Docker Desktop first. - - Then run: - - `just compose-up-grpc-server` +#### Server-based: Run the REST and gRPC servers Natively, With Docker Compose And Kubernetes! - `just compose-up-rest-server` +- **Natively** (without Docker) - - **Kubernetes** (It uses local images to show k8s for local) - Install and start Colima with Kubernetes enabled: - ```bash - brew install colima - colima start --cpu 2 --memory 4 --kubernetes - ``` - When you run colima start --kubernetes, Colima automatically: - - Creates a new Docker context named "colima" - Creates a new Kubernetes context - Sets both contexts as current/active - Updates ~/.docker/config.json for Docker context - Updates ~/.kube/config for Kubernetes context - - List all available Docker and Kubernetes contexts: - ```bash - docker context list - kubectl config get-contexts - ``` - - Verify your Docker and Kubernetes context: - ```bash - docker context show - kubectl config current-context - ``` - Docker also stores context information in ~/.docker/config.json and Kubernetes stores context information in ~ - /.kube/config. - - You can always switch back to Colima's context when needed using: - ```bash - docker context use colima - kubectl config use-context colima - ``` +`just run-drive-deposits-grpc-server` - Install Helm if not already installed: +`just run-drive-deposits-rest-grpc-gateway-server` - ```bash - brew install helm - ``` +- **Docker Compose** + Start Docker Desktop first. - Add and update the nginx-ingress Helm repository: +Then run: - ```bash - helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx - helm repo update - helm repo list - ``` +`just compose-up-grpc-server` - Install the nginx-ingress controller: +`just compose-up-rest-server` - ```bash - helm install ingress-nginx ingress-nginx/ingress-nginx - ``` +- **Kubernetes** (It uses local images to show k8s for local) + Install and start Colima with Kubernetes enabled: - Monitor the nginx ingress controller logs: +```bash +brew install colima +colima start --cpu 2 --memory 4 --kubernetes +``` - ```bash - kubectl logs -l app.kubernetes.io/name=ingress-nginx -f - ``` +When you run colima start --kubernetes, Colima automatically: - Only if using ingress controller: - Add the domain to your /etc/hosts: +Creates a new Docker context named "colima" +Creates a new Kubernetes context +Sets both contexts as current/active +Updates ~/.docker/config.json for Docker context +Updates ~/.kube/config for Kubernetes context - ```bash - echo "127.0.0.1 api.drivedeposits.local" | sudo tee -a /etc/hosts - ``` +List all available Docker and Kubernetes contexts: - Create AWS credentials secret for the gRPC server: +```bash +docker context list +kubectl config get-contexts +``` - ```bash - kubectl create secret generic aws-credentials \ - --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ - --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ - --from-literal=AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION - ``` +Verify your Docker and Kubernetes context: - To verify the secret: + ```bash + docker context show + kubectl config current-context + ``` - ```bash - kubectl get secret aws-credentials -o json - ``` +Docker also stores context information in ~/.docker/config.json and Kubernetes stores context information in ~ +/.kube/config. - Deploy the services: +You can always switch back to Colima's context when needed using: - ```bash - just k8s-grpc-server - just k8s-rest-server - ``` + ```bash + docker context use colima + kubectl config use-context colima + ``` - The REST API will now be accessible at http://api.drivedeposits.local +Install Helm if not already installed: -#### Test microservices integration + ```bash + brew install helm + ``` -In justfile set -for native and docker-compose: -rest_gateway_server_host := "http://localhost:3000" -And -for k8s with ingress: -rest_gateway_server_host := "http://api.drivedeposits.local" +Add and update the nginx-ingress Helm repository: -Send an HTTP request to the REST gateway server, which communicates with other microservices: + ```bash + helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx + helm repo update + helm repo list + ``` -`just post-calculate-portfolio-valid` +Install the nginx-ingress controller: -This command tests the full microservices pipeline, from the REST gateway to the gRPC server, through to AWS EventBridge -and Lambda functions. + ```bash + helm install ingress-nginx ingress-nginx/ingress-nginx + ``` -Follow up with the [Data population](#data-population) section to see how to query data across the microservices. +Monitor the nginx ingress controller logs: -#### Test AWS Lambda microservice directly + ```bash + kubectl logs -l app.kubernetes.io/name=ingress-nginx -f + ``` -`just aws-invoke-drive-deposits-event-rules-lambda` +Only if using ingress controller: +Add the domain to your /etc/hosts: -[Back to Table of Contents](#table-of-contents) + ```bash + echo "127.0.0.1 api.drivedeposits.local" | sudo tee -a /etc/hosts + ``` -### Hybrid Integration Testing Tool +Create AWS credentials secret for the gRPC server: -#### Efficient Command for Synchronous Calculation Flow and Asynchronous AWS Serverless Event Testing +```bash +kubectl create secret generic aws-credentials \ + --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ + --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \ + --from-literal=AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION +``` -The command drive-deposits-check-cmd is a powerful hybrid integration testing tool that bridges both synchronous and -asynchronous aspects of the system. It efficiently mimics the synchronous flow of REST and gRPC servers while -interacting with asynchronous EventBridge components, all without the need for full server deployment. -Key features: +To verify the secret: -* Performs identical type transformations as REST and gRPC servers -* Enables rapid calculation validation and event routing verification -* Sends calculations to EventBridge for comprehensive sanity testing -* Validates event routing to appropriate destinations (log groups, AWS Lambda functions, DynamoDB) -* Allows developers to verify end-to-end flow of calculations, event handling, and data persistence +```bash +kubectl get secret aws-credentials -o json +``` -Execute the tool with: +Deploy the services: -`just run-drive-deposits-check-cmd-valid-send-events` + ```bash + just k8s-grpc-server + just k8s-rest-server + ``` -This streamlined approach significantly enhances development efficiency and system reliability testing. +The REST API will now be accessible at http://api.drivedeposits.local -###### Alias +#### Test microservices integration -.cargo/config.toml has alias for command line [drive-deposits-check-cmd](drive-deposits-check-cmd) so can be run using -`cargo ddcheck` For help see `cargo ddcheck -- --help` +In justfile set -[Back to Table of Contents](#table-of-contents) +for native and docker-compose: -### Running Tests +```justfile +rest_gateway_server_host := "http://localhost:3000" +``` -##### Integration tests +And -`just test-intg` +for k8s with ingress: -##### Unit and Integration tests +```justfile +rest_gateway_server_host := "http://api.drivedeposits.local" +``` -`just test` +Send an HTTP request to the REST gateway server, which communicates with other microservices: -##### End-to-End tests +`just post-calculate-portfolio-valid` -`just test-e2e` +This command tests the full microservices pipeline, from the REST gateway to the gRPC server, through to AWS EventBridge +and Lambda functions. -[Back to Table of Contents](#table-of-contents) +Follow up with the [Data population](#data-population) section to see how to query data across the microservices. -### Data population +#### Test AWS Lambda microservice directly -#### AWS Populated with Basic Data for Queries Lambda +`just aws-invoke-drive-deposits-event-rules-lambda` -##### Using REST gateway and gRPC Servers: start +#### Data population -`just run-drive-deposits-grpc-server` +##### AWS Populated with Basic Data for Queries Lambda -`just run-drive-deposits-rest-grpc-gateway-server` +See [Deployment of Microservices](#deployment-of-microservices) section for deploying the microservices and AWS +resources. -##### send curl post requests to populate +###### send curl post requests to populate `just post-calculate-portfolio-valid` @@ -492,13 +500,13 @@ This streamlined approach significantly enhances development efficiency and syst `just post-calculate-portfolio-valid-greater-amount` -##### Alternatively, Using check command without servers +###### Alternatively, Using check command without servers `just run-drive-deposits-check-cmd-valid-send-events` [Back to Table of Contents](#table-of-contents) -### Querying with custom domain +#### Querying with custom domain Successfully configured custom domain (https://api-queries.drivedeposits.drinnovations.us) for API Gateway so with some existing data queries can be tried @@ -530,6 +538,48 @@ in [justfile](justfile). Adjusted `justfile` with a test portfolio UUID for quer [Back to Table of Contents](#table-of-contents) +### Running Tests + +##### Integration tests + +`just test-intg` + +##### Unit and Integration tests + +`just test` + +##### End-to-End tests + +`just test-e2e` + +##### Hybrid Integration Testing Tool + +###### Efficient Command for Synchronous Calculation Flow and Asynchronous AWS Serverless Event Testing + +The command drive-deposits-check-cmd is a powerful hybrid integration testing tool that bridges both synchronous and +asynchronous aspects of the system. It efficiently mimics the synchronous flow of REST and gRPC servers while +interacting with asynchronous EventBridge components, all without the need for full server deployment. +Key features: + +* Performs identical type transformations as REST and gRPC servers +* Enables rapid calculation validation and event routing verification +* Sends calculations to EventBridge for comprehensive sanity testing +* Validates event routing to appropriate destinations (log groups, AWS Lambda functions, DynamoDB) +* Allows developers to verify end-to-end flow of calculations, event handling, and data persistence + +Execute the tool with: + +`just run-drive-deposits-check-cmd-valid-send-events` + +This streamlined approach significantly enhances development efficiency and system reliability testing. + +###### Alias + +.cargo/config.toml has alias for command line [drive-deposits-check-cmd](drive-deposits-check-cmd) so can be run using +`cargo ddcheck` For help see `cargo ddcheck -- --help` + +[Back to Table of Contents](#table-of-contents) + ### Development Tool: cargo lambda Following is convenience so that in development can iterate faster when skipping sam resources: diff --git a/compose.grpc.yaml b/compose.grpc.yaml index c488944..27afde7 100644 --- a/compose.grpc.yaml +++ b/compose.grpc.yaml @@ -1,3 +1,4 @@ +name: deploy services: drive-deposits-grpc-server: build: diff --git a/compose.rest.yaml b/compose.rest.yaml index e05ea8d..a254955 100644 --- a/compose.rest.yaml +++ b/compose.rest.yaml @@ -1,3 +1,4 @@ +name: deploy services: drive-deposits-rest-gateway-server: build: diff --git a/grpc-server.yaml b/grpc-server.yaml index cec560f..e81ae3e 100644 --- a/grpc-server.yaml +++ b/grpc-server.yaml @@ -52,4 +52,4 @@ spec: - protocol: TCP port: 50052 targetPort: 50052 - type: ClusterIP \ No newline at end of file + type: ClusterIP diff --git a/media/portfolio-request-and-response.png b/media/portfolio-request-and-response.png new file mode 100644 index 0000000..18bb15a Binary files /dev/null and b/media/portfolio-request-and-response.png differ diff --git a/media/valid-portfolio-request.png b/media/valid-portfolio-request.png new file mode 100644 index 0000000..ccd5e5c Binary files /dev/null and b/media/valid-portfolio-request.png differ