Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exports Entity for awsemfexporter plugins on PutLogEvent calls #233

Merged
merged 45 commits into from
Sep 20, 2024

Conversation

dchappa
Copy link

@dchappa dchappa commented Sep 9, 2024

Description: Exports Entity for awsemfexporter on PutLogEvent calls

.

We now add Entity as part of the StreamKey to pass the unique Identity to the proper Log Group and Log Stream. As a result, the pusher map is now bounded (currently 1000) using an lru cache to prevent potential memory exhaustion with the added cardinality.

Testing: Unit testing was added. Manual testing was done on personal EC2 instance and EKS Cluster, using a basic Application Signals instance on the cloudwatch agent.
On EC2

2024-09-09T01:23:16Z I! {"caller":"cwlogs@v0.103.0/pusher.go:144","msg":"entity: {\n  Attributes: {\n    Namespace: \"default\"\n  },\n  KeyAttributes: {\n    Name: \"dinakars-candy-service\",\n    Environment: \"dinakars-candy-environment\",\n    Type: \"Service\"\n  }\n}","kind":"exporter","data_type":"metrics","name":"awsemf/application_signals"}

On EKS, with some added debug lines

{"caller":"awsemfexporter@v0.103.0/metric_translator.go:185","msg":"all resource attributes","kind":"exporter","data_type":"metrics","name":"awsemf/application_signals","attrs":{"aws.entity.k8s.cluster.name":"compass-testing","aws.entity.k8s.namespace.name":"default","aws.entity.k8s.node.name":"ip-192-168-19-139.ec2.internal","aws.entity.k8s.workload.name":"insurance-service-python","aws.entity.service.name":"insurance-service-python","aws.log.group.names":"/aws/containerinsights/compass-testing/application","cloud.account.id":"642135535989","cloud.availability_zone":"us-east-1b","cloud.platform":"aws_eks","cloud.provider":"aws","cloud.region":"us-east-1","ec2.tag.aws:autoscaling:groupName":"eks-ng-4f54b450-00c90496-ec2b-b7dd-e182-03d403faffce","ec2.tag.kubernetes.io/cluster/compass-testing":"owned","host.id":"i-0eaad481ab2e7bff2","host.image.id":"ami-0f77a31aa55644aa2","host.name":"ip-192-168-19-139.ec2.internal","host.type":"m5.large","k8s.container.name":"insurance-service-python","k8s.deployment.name":"insurance-service-python","k8s.namespace.name":"default","k8s.node.name":"ip-192-168-19-139.ec2.internal","k8s.pod.name":"insurance-service-python-67b88dcdcd-klxwh","k8s.replicaset.name":"insurance-service-python-67b88dcdcd","service.name":"insurance-service-python","service.version":"latest","telemetry.auto.version":"0.4.0-aws","telemetry.sdk.language":"python","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.25.0"}}

2024-09-20T03:37:30Z I! {"caller":"cwlogs@v0.103.0/pusher.go:295","msg":"Making PLE call.","kind":"exporter","data_type":"metrics","name":"awsemf/application_signals","NumOfLogEvents":18,"LogEventsSize":29.3798828125,"Time":0,"Entity":"{\n  Attributes: {\n    Cluster: \"compass-testing\",\n    Namespace: \"default\",\n    Workload: \"insurance-service-python\",\n    Node: \"ip-192-168-19-139.ec2.internal\"\n  },\n  KeyAttributes: {\n    Name: \"insurance-service-python\"\n  }\n}"}

I currently do not have access to the Entity experience, but can confirm that metrics continue to publish
image

@dchappa dchappa self-assigned this Sep 9, 2024
@dchappa dchappa requested a review from mxiamxia as a code owner September 9, 2024 01:29
@dchappa dchappa removed the request for review from mxiamxia September 9, 2024 01:43
exporter/awsemfexporter/emf_exporter.go Outdated Show resolved Hide resolved
internal/aws/cwlogs/pusher.go Show resolved Hide resolved
exporter/awsemfexporter/bounded_pusher_map.go Outdated Show resolved Hide resolved
exporter/awsemfexporter/bounded_pusher_map.go Outdated Show resolved Hide resolved
exporter/awsemfexporter/bounded_pusher_map.go Outdated Show resolved Hide resolved
Copy link

@nathalapooja nathalapooja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, assuming the least used pusher, will not have any logs pending publishing to PLE

@dchappa dchappa merged commit 299ef87 into aws-cwa-dev Sep 20, 2024
142 of 146 checks passed
@dchappa dchappa deleted the entityExport branch September 20, 2024 14:28
@dchappa dchappa removed the request for review from zhihonl September 20, 2024 14:31
movence added a commit to movence/opentelemetry-collector-contrib that referenced this pull request Oct 1, 2024
@movence movence mentioned this pull request Oct 1, 2024
movence added a commit that referenced this pull request Oct 1, 2024
* Revert "Exports Entity for awsemfexporter plugins on PutLogEvent calls (#233)"

This reverts commit 299ef87.

* Revert "Adds sdk changes necessary for adding Entity field to CW Logs PutLogEvent API (#231)"

This reverts commit 69bd759.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants