aws-iot-sensors-infra

Terraform IaC for the AWS IoT Sensors project.

Overview

This project creates a serverless architecture that ingests IoT sensors data from STM32 cards and publishes the data through a REST API. Data is processed by IoT Core, an IoT Rule and a Lambda function that records messages to a DynamoDB table. The IoT Rule processes data published to a specific MQTT topic, /motion_sensor_data`, where ` represents the device ID.

A REST API in API Gateway allows to query the database through HTTPS. An empty S3 bucket is also configured as a static website, so that the aws-iot-sensors-front web app can be published and display real time graphs by querying the API.

A DynamoDB stream, a Lambda function and a Kinesis Data Firehose are used to archive old DynamoDB items in a S3 bucket. This uses DynamoDB’s TTL feature.

The project is written with reusable modules and makes it easy to extend the architecture to record, publish and archive messages from other types of sensors of the STM32 cards, such as the environment sensors (temperature, humidity…​).

Architecture diagram

Figure 1. Architecture in the AWS cloud. The cloud infrastructure is entirely serverless.

Requirements

You will need these tools:

• Git LFS

• Terraform

In addition, follow the steps in the sections below to configure:

• An AWS account

• The AWS CLI

• An STM32 card that sends, over WiFi, MQTT messages to a {device_id}/motion_sensor_data topic. The project was tested with a B-U585I-IOT02A card.

All instructions in this document were tested on Ubuntu 22.04.

Open an AWS account

Sign up for an AWS account at https://aws.amazon.com/free/. You will need to provide a credit card number. This creates your root user.

Install and configure the AWS CLI

Follow these steps to install the AWS CLI.

Once installed, you need to configure the CLI to use your AWS account. For this, we will create a non-root user with administrator access, create an API key and configure the CLI to use that API key.

1. Sign in to your AWS account at https://aws.amazon.com/ with your root user email and password.

2. In the search bar at the top, search for "IAM" (Identity and Access Management). Click on IAM.

3. In the left pane of the IAM console, navigate to Access management › Users and click on Add users. In the form, choose a username, check the "Provide user access to the AWS Management Console" box, then "I want to create an IAM user". Choose a password and click Next. For the permissions, choose "Attach policies directly". Search for the AdministratorAccess policy, select it and click Next. In the next window, click Create user.

4. Go back to the users list in Access management › Users, click on the new user, open the "Security credentials" tab and click the Create access key button. In the new window, select "Command Line Interface (CLI)", tick the "I understand […​]" box and click Next. Click Create access key. Take note of the generated access key and secret access key.

5. In a terminal, type aws configure. When prompted, provide the access key and the secret access key. For the default region, type eu-west-3. For the default output format, you can keep the default value.

To test that the CLI can make API calls to AWS and access your account, you can run a test command such as the one below, which returns details on your account and current user:

$ aws sts get-caller-identity
{
    "UserId": "AI?????????????????5N",
    "Account": "12????????89",
    "Arn": "arn:aws:iam::12????????89:user/nicolas"
}

Configure the STM32 card to send data to AWS IoT Core

In its default configuration, a B-U585I-IOT02A card is capable of sending MQTT messages to an AWS IoT endpoint, simply by configuring it to connect to your AWS account. This is done by running an ST-provided script with the card plugged to a computer with USB. The script interacts with the card through a serial communication.

1. Add your UNIX account to the dialout group in order to be able to use serial ports:

   1. Add yourself to the group (adapt the username):

      $ sudo addgroup nicolas dialout

   2. Reboot your computer. You can check you are in the dialout group by listing your groups with the groups command.

2. Clone the FreeRTOS iot-reference-stm32u5 project:

   git clone https://github.com/FreeRTOS/iot-reference-stm32u5.git

3. Configure a Python environment:

   $ cd iot-reference-stm32u5/tools
   $ python3 -m venv .venv
   $ source .venv/bin/activate
   (.venv)$ pip install -r requirements.txt

4. Configure the STM32 card:

   1. Plug the card with a USB cable to the computer.

   2. Execute the provision.py script. This script configures the card to connect to a WiFi network, and registers the card in your AWS account. The registration involves telling the card what your IoT endpoint (URL) is, registering the device as an AWS IoT Thing in your account, and generating an SSL certificate so that the card has the private key and your AWS account has the public certificate that AWS will use to authenticate and identify your card as a certain device.

      (.venv)$ python provision.py --aws-profile default --thing-name stm32_1 --wifi-ssid SSID --wifi-credential WIFI_PASSWORD

      In the command above, replace SSID and WIFI_PASSWORD by your WiFi network parameters. The output for a successful registration looks like:

      Target device path: /dev/ttyACM0
      Connecting to target...
      [ INFO ] Found credentials in shared credentials file: ~/.aws/credentials (credentials.py:load)
      Commiting target configuration...
      Generating a new public/private key pair
      Generating a self-signed Certificate
      Attaching thing: stm32_1 to principal: arn:aws:iot:eu-west-3:12????????89:cert/2b????????????????????????????????????????????????????????????0d
      [ INFO ] Existing policy "AllowAllDev" was not found. Creating it... (provision.py:create_policy)
      Attaching the "AllowAllDev" policy to the device certificate.
      Importing root ca certificate: "Starfield Services Root Certificate Authority - G2"
      Provisioning process complete. Resetting target device...

To test the STM32 card configuration, navigate, in your AWS account, to the IoT Core AWS service using the top search bar, then to Test › MQTT test client › Subscribe to a topic, and subscribe to +/motion_sensor_data. This should display within a fraction of a second JSON messages such as this one:

{
  "acceleration_mG": {
    "x": 1,
    "y": -1,
    "z": 1009
  },
  "gyro_mDPS": {
    "x": 140,
    "y": -210,
    "z": -560
  },
  "magnetometer_mGauss": {
    "x": 117,
    "y": 229,
    "z": -13
  }
}

Tip	You can now unplug the card. Plugging it again to any computer or even just to a power source will start sending data to AWS IoT Core.

Warning

When reconfiguring a previously configured card with the provision.py script, errors may occur if the card is still connected over WiFi. A solution to this is to turn off the WiFi network before running the script.

Usage

Clone the aws-iot-sensors-infra repository and cd into it.

Configure the environment

Create a terraform.tfvars file at the root of the repository. Edit it to alter the default configuration. The list of variables can be found in variables.tf.

For example, to change the items TTL in DynamoDB to seven days, add this line to terraform.tfvars:

dynamodb_item_ttl = 604800

Terraform deployment

To deploy the resources, execute:

$ terraform init
$ terraform apply

Type "yes" when asked to accept the solution. The command prints useful information such as the API and Web URLs.

Note	The command above only works if you have already configured your AWS CLI with aws configure.

To delete the resources, run:

$ terraform destroy

Tip	The architecture may induce small storage, data ingestion and retrieval costs. It is good to destroy all resources when they are not needed anymore.

Tip	In case of errors related to non-empty buckets, you can delete the data in the buckets (aws s3 rm --recursive s3://BUCKET_NAME) and run terraform destroy again.

Next steps

Release an API version

The deployment of the API Gateway REST API is not managed by this Terraform project. After initial deployment or changes to the architecture, run the command provided in the deploy_command output variable that is printed by terraform apply or terraform output.

If everything went well, querying the /v1/measurements/motion/{device_id}/recent URL should return data that looks like this:

{
  "Count": 604,
  "Items": [
    {
      "payload": {
        "M": {
          "acceleration_mG_z": {
            "N": "1009"
          },
          "magnetometer_mGauss_x": {
            "N": "115"
          },
          "gyro_mDPS_x": {
            "N": "140"
          },
          "magnetometer_mGauss_y": {
            "N": "225"
          },
          "acceleration_mG_x": {
            "N": "1"
          },
          "gyro_mDPS_y": {
            "N": "-210"
          },
          "magnetometer_mGauss_z": {
            "N": "-13"
          },
          "acceleration_mG_y": {
            "N": "-1"
          },
          "gyro_mDPS_z": {
            "N": "-490"
          }
        }
      },
      "ttl": {
        "N": "1682092119.359"
      },
      "device": {
        "S": "stm32_1"
      },
      "timestamp": {
        "N": "1682092089.359"
      }
    },
    ...
  ],
  "ScannedCount": 604
}

Deploy the static website

The architecture comprises an S3 Bucket that can hold a static website. You may want to upload static files in this bucket for a front app that queries the API. You can use the demo such as aws-iot-sensors-front project to generate such a site. See aws-iot-sensors-front's documentation for more details.

Exercise

By default, the STM32 cards publish MQTT messages to /env_sensor_data` topics, with temperature and other data, in addition to `/motion_sensor_data. Also, the aws-iot-sensors-front companion project has a page that displays graphs from temperature sensors. But it currently queries an API URL that does not exist!

Try to modify aws-iot-sensors-infra to record the temperature data and add the missing API endpoint that returns the recent temperature records. Then, deploy the changes by running terraform init and terraform apply, and then by deploying an API Gateway stage.