Mirroring auto-created/destroyed for EC2 instances #45

chelma · 2023-05-03T22:28:03Z

Description

Set up an EventBridge Rule to listen for EC2 instance stops and starts and direct them to the EventListener Lambda which converts them into Create/Destroy ENI Mirroring Events
Added an ECS-on-EC2 cluster to the Demo Traffic Stacks which can be used to test the new rule.
There's a potential race condition in the code. We catch/process EC2 instance state change events, which only provide the instance ID. We then use the instance ID to figure out which ENIs to stop mirroring by making an ec2:DescribeInstances API call. The issues is that terminated EC2 instances don't have attached ENIs. We catch/process the shutting-down event which precedes terminated, and the ENIs are still available then, but if there's an unexpected delay in processing it's possible that we might make our ec2:DescribeInstances call after the ENIs have been removed from the instance, at which point we won't know which ENI to tear mirroring down for.
REMAINING WORK: Update the README to describe our Event Processing system

Tasks

Testing

Added unit tests
Manually added/removed capacity from the Demo Traffic Stack's new ASG to trigger EC2 Instance lifecycle events. Lambda invocation logs below.

AwsEventListener Lambda: EC2 Instance Running

START RequestId: e03d2b90-f030-4de8-ab3e-23650e05c68d Version: $LATEST
Event:
{
    "version": "0",
    "id": "1f8939b9-6ce5-510c-13d7-a2821a5cc5c1",
    "detail-type": "EC2 Instance State-change Notification",
    "source": "aws.ec2",
    "account": "XXXXXXXXXXXX",
    "time": "2023-05-03T21:31:26Z",
    "region": "us-east-2",
    "resources": [
        "arn:aws:ec2:us-east-2:XXXXXXXXXXXX:instance/i-092528d09663600ee"
    ],
    "detail": {
        "instance-id": "i-092528d09663600ee",
        "state": "running"
    }
}

Pulling context from Lambda Environment Variables...
Event Bus ARN: arn:aws:events:us-east-2:XXXXXXXXXXXX:event-bus/MyClusterCaptureNodesClusterBus8101CEAA
Cluster Name: MyCluster
VPC ID: vpc-069bca118708c0fcb
Traffic Filter ID: tmf-0f88f3ec13c38bd6b
Mirror VNI: 42
Parsing AWS Service Event...
Event Type: EC2 Instance Running
Processing EC2 Instance: i-092528d09663600ee
Found credentials in environment variables.
ENIs:
[
{
    "vpc_id": "vpc-069bca118708c0fcb",
    "subnet_id": "subnet-0aeeeea40ba9b0406",
    "eni_id": "eni-010c2ec47525cf2aa",
    "eni_type": "interface"
}
]
Preparing CreateEniMirrorEvent: 
{
    "source": "arkime",
    "detail_type": "CreateEniMirror",
    "details": {
        "cluster_name": "MyCluster",
        "vpc_id": "vpc-069bca118708c0fcb",
        "subnet_id": "subnet-0aeeeea40ba9b0406",
        "eni_id": "eni-010c2ec47525cf2aa",
        "eni_type": "interface",
        "traffic_filter_id": "tmf-0f88f3ec13c38bd6b",
        "vni": 42
    }
}

Initiating creation of mirroring session(s) for 1 ENI(s)
Found credentials in environment variables.
END RequestId: e03d2b90-f030-4de8-ab3e-23650e05c68d
REPORT RequestId: e03d2b90-f030-4de8-ab3e-23650e05c68d  Duration: 4026.61 ms    Billed Duration: 4027 ms    Memory Size: 128 MB Max Memory Used: 85 MB  Init Duration: 291.43 ms

CreateEniMirroring Lambda

START RequestId: 2bfb9fc3-c613-448c-a1ab-b908f660a5b7 Version: $LATEST
Event:
{
    "version": "0",
    "id": "28c5223d-8808-71de-89fa-1a6f7e846d3d",
    "detail-type": "CreateEniMirror",
    "source": "arkime",
    "account": "XXXXXXXXXXXX",
    "time": "2023-05-03T21:31:31Z",
    "region": "us-east-2",
    "resources": [],
    "detail": {
        "cluster_name": "MyCluster",
        "vpc_id": "vpc-069bca118708c0fcb",
        "subnet_id": "subnet-0aeeeea40ba9b0406",
        "eni_id": "eni-010c2ec47525cf2aa",
        "eni_type": "interface",
        "traffic_filter_id": "tmf-0f88f3ec13c38bd6b",
        "vni": 42
    }
}
Starting Traffic Mirroring Session creation process for ENI eni-010c2ec47525cf2aa
Found credentials in environment variables.
Confirmed SSM Param does not exist for ENI eni-010c2ec47525cf2aa
Found credentials in environment variables.
Creating Mirroring Session...
Found credentials in environment variables.
Creating SSM Parameter: /arkime/clusters/MyCluster/vpcs/vpc-069bca118708c0fcb/subnets/subnet-0aeeeea40ba9b0406/enis/eni-010c2ec47525cf2aa
Found credentials in environment variables.
Found credentials in environment variables.
END RequestId: 2bfb9fc3-c613-448c-a1ab-b908f660a5b7
REPORT RequestId: 2bfb9fc3-c613-448c-a1ab-b908f660a5b7  Duration: 8040.07 ms    Billed Duration: 8041 ms    Memory Size: 128 MB Max Memory Used: 116 MB

AwsEventListener Lambda: EC2 Instance Shutting Down

START RequestId: 1687de52-72b4-4811-8930-f92a44d68f04 Version: $LATEST
Event:
{
    "version": "0",
    "id": "b48beeb8-eb10-1a28-e5ef-69867b242ff9",
    "detail-type": "EC2 Instance State-change Notification",
    "source": "aws.ec2",
    "account": "XXXXXXXXXXXX",
    "time": "2023-05-03T22:07:26Z",
    "region": "us-east-2",
    "resources": [
        "arn:aws:ec2:us-east-2:XXXXXXXXXXXX:instance/i-0bd5e40d8037dfb63"
    ],
    "detail": {
        "instance-id": "i-0bd5e40d8037dfb63",
        "state": "shutting-down"
    }
}

Pulling context from Lambda Environment Variables...
Event Bus ARN: arn:aws:events:us-east-2:XXXXXXXXXXXX:event-bus/MyClusterCaptureNodesClusterBus8101CEAA
Cluster Name: MyCluster
VPC ID: vpc-069bca118708c0fcb
Traffic Filter ID: tmf-0f88f3ec13c38bd6b
Mirror VNI: 43
Parsing AWS Service Event...
Event Type: EC2 Instance Shutting Down
Processing EC2 Instance: i-0bd5e40d8037dfb63
Found credentials in environment variables.
ENIs:
[
{
    "vpc_id": "vpc-069bca118708c0fcb",
    "subnet_id": "subnet-0aeeeea40ba9b0406",
    "eni_id": "eni-05198aa9f8d5a9787",
    "eni_type": "interface"
}
]
Preparing DestroyEniMirrorEvent: 
{
    "source": "arkime",
    "detail_type": "DestroyEniMirror",
    "details": {
        "cluster_name": "MyCluster",
        "vpc_id": "vpc-069bca118708c0fcb",
        "subnet_id": "subnet-0aeeeea40ba9b0406",
        "eni_id": "eni-05198aa9f8d5a9787"
    }
}

Initiating destruction of mirroring session(s) for 1 ENI(s)
Found credentials in environment variables.
END RequestId: 1687de52-72b4-4811-8930-f92a44d68f04
REPORT RequestId: 1687de52-72b4-4811-8930-f92a44d68f04  Duration: 3706.60 ms    Billed Duration: 3707 ms    Memory Size: 128 MB Max Memory Used: 85 MB  Init Duration: 233.52 ms

DestroyEniMirroring Lambda

START RequestId: 5c291c2a-fa8b-4c8d-b447-337ac6de1dbf Version: $LATEST
Event:
{
    "version": "0",
    "id": "a92dac89-8ed0-4d5f-6a11-fac6b351a533",
    "detail-type": "DestroyEniMirror",
    "source": "arkime",
    "account": "XXXXXXXXXXXX",
    "time": "2023-05-03T22:07:30Z",
    "region": "us-east-2",
    "resources": [],
    "detail": {
        "cluster_name": "MyCluster",
        "vpc_id": "vpc-069bca118708c0fcb",
        "subnet_id": "subnet-0aeeeea40ba9b0406",
        "eni_id": "eni-05198aa9f8d5a9787"
    }
}

Found credentials in environment variables.
Removing mirroring session for eni eni-05198aa9f8d5a9787: tms-03cefa0df79df536b...
Found credentials in environment variables.
Deleting SSM parameter for ENI eni-05198aa9f8d5a9787: /arkime/clusters/MyCluster/vpcs/vpc-069bca118708c0fcb/subnets/subnet-0aeeeea40ba9b0406/enis/eni-05198aa9f8d5a9787
Found credentials in environment variables.
Found credentials in environment variables.
END RequestId: 5c291c2a-fa8b-4c8d-b447-337ac6de1dbf
REPORT RequestId: 5c291c2a-fa8b-4c8d-b447-337ac6de1dbf  Duration: 6190.81 ms    Billed Duration: 6191 ms    Memory Size: 128 MB Max Memory Used: 100 MB Init Duration: 268.40 ms

* Set up an EventBridge Rule to listen for EC2 instance stops and starts and direct them to the EventListener Lambda which converts them into Create/Destroy ENI Mirroring Events Signed-off-by: Chris Helma <chelma+github@amazon.com>

awick · 2023-05-03T22:36:01Z

cdk-lib/traffic-gen-sample/traffic-gen-stack.ts

        });

-        const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef', {
+        const fargateTaskDef = new ecs.FargateTaskDefinition(this, 'TaskDef', {


I always wonder does the id passed to these constructors matter? In the variable renaming I see you updated some (SsmKey), but there are some that you didn't. (TaskDef, Service)

The ID ("TaskDef") is used in the name of the AWS Resource(s) associated with the CDK Construct you're instantiating. I can think of three considerations:

Making sure your name isn't too long that it overflows the available character limit

Making sure your name is sufficiently descriptive

Making sure your name doesn't collide with any other name

That said - you're really free to pass in whatever you like. Another considerations is that, typically, changing the ID will require the AWS Resource(s) associated with the Construct to be torn down and recreated in order for the name switch to take effect.

Mirroring auto-created/destroyed for EC2 instances

6b1f650

* Set up an EventBridge Rule to listen for EC2 instance stops and starts and direct them to the EventListener Lambda which converts them into Create/Destroy ENI Mirroring Events Signed-off-by: Chris Helma <chelma+github@amazon.com>

chelma added the Capture Resilience Work to make traffic capture more resilient to changes in load, configuration, and sources label May 3, 2023

chelma requested review from awick and 31453 May 3, 2023 22:28

awick approved these changes May 3, 2023

View reviewed changes

chelma merged commit 2d02685 into main May 3, 2023

chelma deleted the ec2-events branch May 3, 2023 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mirroring auto-created/destroyed for EC2 instances #45

Mirroring auto-created/destroyed for EC2 instances #45

chelma commented May 3, 2023

awick May 3, 2023

chelma May 3, 2023

chelma May 3, 2023

Mirroring auto-created/destroyed for EC2 instances #45

Mirroring auto-created/destroyed for EC2 instances #45

Conversation

chelma commented May 3, 2023

Description

Tasks

Testing

awick May 3, 2023

Choose a reason for hiding this comment

chelma May 3, 2023

Choose a reason for hiding this comment

chelma May 3, 2023

Choose a reason for hiding this comment