Skip to content

Latest commit

 

History

History
543 lines (362 loc) · 20.5 KB

README.md

File metadata and controls

543 lines (362 loc) · 20.5 KB
Raw

MQTT — Simple Pub/Sub using Lambda

Tables of Contents:

Overview

Welcome to mini workshop about MQTT using AmazonMQ - Lambda Function - ReactJS!

In this mini workshop, you will learn both about publishing and subscribing messages using MQTT. You will also learn how to setup/deploy web-app and lambda function in AWS. Please take a look the architecture below.

The content of this mini workshop may be updated and if you have questions or find issues in this mini workshop, please file them as an Issue.

Project Structure

mqtt-simple-pubsub/
├─ functions/
├─ web/
├─ .gitignore
├─ architecture.png
├─ LICENSE
├─ README.md
  • functions/ contains lambda function code for publishing messages to MQTT.
  • web/ contains source code of web-app build with ReactJS. This web-app works as a subcriber and will displays the data sent by the publisher.
  • architecture.png is an overview of the resources to be deployed.
  • README.md contains guide for this mini workshop.

Requirements

Before starting this mini workshop, the following runtime/tools must be met and configured properly.

  • Active AWS Account.
  • NodeJS v16 or latest.
  • AWS CLI version 2.
  • (optional) OS based on Linux.

    Note
    Build script for package lambda function code and it's dependecies require Linux/Unix shell to operate. If you are using an OS other than Linux and/or your device doesn't support Linux shell commands you can customize this build script to make sure it runs properly.

AWS Resources

Some of the services from AWS that are used in this mini workshop are as follows:

Setup Environment

Setup Infrastructure

Note
If the settings/configurations are not specified in this guide, you can leave them as default or you can specify the values with your own.

  1. Select AWS region.

  2. Create VPC with following specifications:

    • Has two or more subnets.
    • Has internet gateway attached/route to public subnets.
    • (optional) At least one NAT Gateways route to private subnets.

      💡 TIP

      Use it when the resource(s) needed internet connection. Note that there is a charge for each NAT gateway.

    • DNS hostnames and DNS resolution must be enabled.

  3. Create first security group with following rules:

    • Allow port 8162 from anywhere
    • Allow port 8883 from anywhere
    • Allow port 61619 from anywhere
    • Allow all traffic/port to anywhere

  4. Create Amazon MQ broker with following spefications:

    • Broker engine: Apache Active MQ
    • Deployment mode: Single-instance broker
    • Storage type: Durability optimized
    • Broker instance type: mq.t3.micro
    • ActiveMQ access: Simple Authentication and Authorization

      Fill username and password with your own. Make sure you store it somewhere else so you don't forget. Alternatively, you can store it in AWS SSM parameter store or AWS Secret Manager.

    • Broker engine version: 5.17.3
    • Access type: Public access
    • VPC and subnets: Select existing VPC and subnet(s)

      Select VPC that you've created before.

    • Security groups: Select existing security groups

      Select first security group that you've created before.

  5. Create second security group with following rules:

    • Allow HTTP/HTTPS port from anywhere
    • (optional) Allow port 3000 from anywhere
    • (optional) Allow SSH port from your IP
    • Allow all traffic/port to anywhere

  6. Create EC2 instance with following configurations:

    For AMI, Instance Type and Storage doesn't have to be exactly the same. For example you can choose instance type that has more large or small than the recommended type.

    • OS/Amazon Machine Image (AMI): Ubuntu 22.04 LTS (Recommended)
    • Instance Type: t4g.small (Recommended)
    • VPC: Select VPC that you've created before.
    • Subnet: Select subnet that you've created before.
    • Auto-assign public IP: true
    • Security group: Select existing security group

      Select second security group that you've created before.

    • Storage size: 10 GiB (Recommended)
    • Storage type: gp3 (Recommended)

Setup Lambda Publisher

Note
If the settings/configurations are not specified in this guide, you can leave them as default or you can specify the values with your own.

  1. Select the same AWS region as the infrastructure that has been deployed.

  2. Create Lambda function with the following configurations:

    • Runtime: Node.js 18.x
    • Memory: 128 MB
    • Handler: publisher.handler
    • Enable VPC

      Select VPC and subnet group that you've created before. For security group you can use default security group or you can create custom security group.

  3. Configure Lambda enviroment variables:

    MQ_HOST=<your-amazonmq-mqtt-endpoint>
MQ_TOPIC=broadcast
MQ_USERNAME=<your-amazonmq-username>
MQ_PASSWORD=<your-amazonmq-password>

    example:

    MQ_HOST="mqtt+ssl:b-123abc-45d-67ef-890123abc.mq.ap-southeast-1.amazonaws.com:8883"
MQ_TOPIC=broadcast
MQ_USERNAME=admin
MQ_PASSWORD=admin1234567

  4. Build and publish code

    • Clone this repository

      git clone https://github.com/zrierc/mqtt-simple-pubsub.git && cd mqtt-simple-pubsub/

    • Navigate to functions/

      cd functions/

    • Install required dependencies

      npm i

      or

      yarn

    • Build your lambda function code and dependencies to zip

      npm run build

      or

      yarn build

    • Publish your zip package to lambda using AWS CLI:

      aws lambda update-function-code \
  --function-name <your-lambda-function-name> \
  --zip-file fileb://function.zip

  5. Test invoke function with the following event:

    {
  "message": "hello from lambda!"
}

    Note
    Feel free to change the message value.

😕 Are you stuck?

Check this common issues:

❌ Failed running build command
  • Make sure npm or yarn already installed on your machine.
  • Make sure zip and unzip already installed on your machine. Click here to read guide how to install zip and unzip in Linux machine.
  • Make sure your machine support Linux/Unix commands to run build script. For Windows user, you can try to install WSL 2 to run Linux on your Windows machine OR you can create custom build script using powershell/cmd. Don't forget to change build script command inside package.json.
⚠ Error when publish package using AWS CLI
  • Make sure you code already built and function.zip in the same directory as functions/ exists.
  • Make sure you already installed AWS CLI on your machine. Click here to read more about how to install the AWS CLI.
  • Setup IAM credentials for authentication. Click here to read more about how to setup credentials.

    Note
    Make sure your IAM credentials has AWSLambda_FullAccess policy.

🧪 Test/invoke your Lambda

You can test/invoke your Lambda function

You can check AWS CloudWatch Logs for the results. Click here lo read more how to access Cloudwatch Logs from Lambda.

⚠ Error can't publish message to Amazon MQ
  • Make sure you set Lambda environment variables. Click here to learn how to configure Lambda environment variables.
  • Use the correct Amazon MQ endpoints inside environment variable.
  • Use the correct credentials (username & password) for authenticate to Amazon MQ.
  • Make sure your Lambda function connected to the same VPC as Amazon MQ. Don't forget to setup your security group rules.

Setup Web-App

Note
If the settings/configurations are not specified in this guide, you can leave them as default or you can specify the values with your own.

  1. Select the same AWS region as the infrastructure that has been deployed

  2. Connect to EC2 instance that already created

    Note
    EC2 gives you several ways to connect to it. Learn more.

  3. Install required packages:

  4. Clone this repository and navigate to web/

    git clone https://github.com/zrierc/mqtt-simple-pubsub.git && cd mqtt-simple-pubsub/

cd web/

  5. Install required dependencies

    npm i

    or

    yarn

  6. Setup enviroment variables

    • Copy .env boilerplate

      cp .env.example .env

    • Fill the required environment variables

      BASE_URL=[http://your-domain]
VITE_MQ_HOST=[wss://your-amazonmq-wss-endpoint:61619]
VITE_MQ_TOPIC=broadcast
VITE_MQ_USERNAME=[your-amazonmq-username]
VITE_MQ_PASSWORD=[your-amazonmq-password]

  7. Build web-app

    npm run build

    or

    yarn build

  8. Start web-app

    You can customize the port number. (e.g., --port 80)

    npm run preview --port 3000

    or

    yarn run preview --port 3000

    Note
    You can also running this web-app behind reverse proxy Nginx or using pm2.

  9. Open/test your web-app by accessing the public IP or public IPv4 DNS via web browsers.

😕 Are you stuck?

Check this common issues:

⚠ Can't access web-app
  • Expose app publicly by adding argument --host 0.0.0.0. For example: 
    yarn run preview --host 0.0.0.0 --port 3000
  • Make sure your web-app running properly and can publicly accessible or if you use Nginx as reverse proxy make sure you configure Nginx properly.
  • Check the security group that attached to EC2 Instance. Make sure the port that used to run the web-app is added to the inbound rules security group.
❌ Data/message on web-app not appears

Check the browser console. If you got an error with message:

WebSocket connection to 'wss://abcd-ef-123-4567890.mq.ap-southeast-1.amazonaws.com:61619/' failed

or

Firefox can’t establish a connection to the server at wss://localhost:61619/.

You probably use the wrong endpoint and authentication in .env file. Please review and edit .env with following rules:

🧪 Testing

  1. Invoke Lambda function with following event:

    Note
    Feel free to change the message value.

    {
  "message": "hello from lambda!"
}
    🎬 Behind the Scene

    • Lambda will receive the event that you sent, get message value and proccess it.

    • After the event being proccessed, Lambda will publish it to topic called broadcast on Amazon MQ using mqtt protocol.

    • Each payload stored in Amazon MQ.

  2. Open the web-app that you've already deployed in web browsers and you will see message that you sent in Lambda.

    💡 TIP

    If data are not shown, invoke Lambda again by repeating step 1.

    🎬 Behind the Scene

    Web-app subcribe topic called broadcast on Amazon MQ using web socket (wss) protocol and display it in tabular form.

    Note
    Since the data are not stored in persistent storage, if you refresh your web browsers the data will lost. However, you can still receive data again by invoking Lambda as in step 1.

Improving Your Web-App

❗❗ First Thing First

To follow this section you MUST complete Setup Environment section and mare sure everything runs well.

Welcome to the improvement section of this mini workshop! Here you will learn how to improve and optimize your web-app for production such as implementing security best practices and adding high availability.

Implement Security & Best Practices

Infrastructure

  1. Implement custom security group to control traffic at instance level.

    Restrict inbound rule by adding only the necessary ports and traffic to the rule. For example, if your web-app not using port 3000 anymore you can remove it from security group. Learn more about security group here.

  2. Implement custom network ACLs to control traffic at the subnet level.

    You can create custom network ACLs for each subnet types. For example, create custom network ACLs for public subnet(s) and private subnet(s). Learn more about network ACLs here.

    Note
    Please remember that network ACLs are stateless, which means that any changes applied to an inbound rule will not be applied to the outbound rule (and vice versa). Also, when you create custom network ACLs by default it will deny all traffic both inbound and outbound traffic.

    There's an associated outbound rule that allows responses to incoming traffic, you can do this by adding ephemeral ports to outbound rule. Only add the required port and traffic to the rule.

  3. Don't make Amazon MQ publicly accessible and put them in private subnet. This ensures that only you (and authorized users) can access it.

Lambda function

  1. Grant only the permissions required to perform a task to your Lambda function role.

    💡 TIP

    For this scenario, you can simply use lambda basic permissions (AWSLambdaBasicExecutionRole) from AWS managed policies.

Web-App

  1. Restrict your security group that attached to your EC2 instance that serve your web-app.

    Restrict inbound rule by adding only the necessary ports and traffic to the rule.

  2. Use HTTPS connection for your web-app.

    To encrypts communications between a client and server you need to add SSL/TLS to your web-app.

    Click here to learn more about setup pm2 with nginx using SSL/LTS.

  3. Use bastion host to connect to your EC2 Instance.

    Bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet. This is provides secure access to your EC2 instance. So, to connect to your EC2 instance that serve your web-app you need to connect bastion host first. Learn more about bastion host here.

    To create bastion host, you can launch EC2 instance in public subnet. For security group, open SSH connection from your IP. Read more about deployment guide bastion host.

  4. Use custom scripts to setup your web-app where possible.

    Use custom bash/shell script to automate your deployment. This can be done using EC2 user data. Simply add your custom bash/shell script to your EC2 user data before it launch. This is useful so you don't need to connect to EC2 directly.

  5. (optional) Put EC2 behind load balancer.

    You can put EC2 instance in private subnet. Use an Application Load Balancer (ALB) in front of it so that users can access the web-app without having to dirrectly access the EC2 instance.

    You can also use HTTPS to your Application Load Balancer (ALB). Learn more.

Clean Up Resources

To minimize the cost, make sure you delete resources used in this workshop when you are finished. Follow the sequence according to the steps to delete the AWS resources:

  1. Delete Amazon MQ broker.

    Click here to read more how to delete Amazon MQ broker.

  2. Delete/terminate EC2 instance.

    Click here to read more how to terminate EC2 instance.

  3. Delete NAT Gateway.

    Step to delete NAT Gateway via AWS Web Console:

    • Access the Amazon VPC console.
    • In the nativation pane, choose NAT Gateways.
    • Select the option button for the NAT Gateway, and then choose Actions, Delete NAT gateway.
    • When prompted, enter delete and then choose delete.

    Note
    The NAT gateway entry might remain visible in the Amazon VPC console for an hour after removal.

    To delete a NAT gateway with the AWS Command Line Interface (AWS CLI), see delete-nat-gateway.

    Warning
    After deleting NAT Gateways, if you no longer need the Elastic IP address that was associated with a public NAT gateway, release the Elastic IP address. Learn more.

  4. Delete VPC.

    Click here to read more how to delete VPC.

    Note
    If you delete a VPC using the Amazon VPC console, VPC components (e.g. Subnets, Security Groups, Route Tables, Internet Gateways, etc) are also deleted.

  5. (optional) Delete credentials that stored in AWS SSM parameter store or AWS Secret Manager.

    If you storing your credentials (username & password) Amazon MQ that stored in AWS SSM parameter store or AWS Secret Manager you can delete it by folllowing these step:

  6. (optional) Delete credentials that associated to AWS CLI.

    Although it costs no money, for security reasons, you can delete credentials used in the AWS CLI if you don't use them anymore.