Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for semantic convention 1_20 and 1_23 #1903

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add support for semantic convention 1_20 and 1_23 #1903

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8269617
fix: Add support for semantic convention 1_20 and 1_23
Sanjalee-NewRelic Feb 7, 2025
555c259
fix: Made relationship files more specific to AWS
Sanjalee-NewRelic Feb 10, 2025
37c7f98
fix: Add Opentelemetry origin
Sanjalee-NewRelic Feb 10, 2025
c84e2f0
fix: Add otel service conditions
Sanjalee-NewRelic Feb 10, 2025
a5ad237
fix: Renamed file
Sanjalee-NewRelic Feb 11, 2025
a032eff
fix: Add aws specific condition for mrmcached
Sanjalee-NewRelic Feb 12, 2025
769822a
Pull latest code from main Merge branch 'main' of https://github.com/…
Sanjalee-NewRelic Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 82 additions & 4 deletions relationships/synthesis/EXT-SERVICE-to-INFRA-MQBROKER.yml
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the name of this file be EXT-SERVICE-to-INFRA-AWSMQBROKER? Or do we anticipate that in the future this file will contain relationship definitions for MQ Brokers other than AWS?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let's keep it scoped to the domain/types involved, we can create another file if we need to extend beyond of AWS.

Copy link
Member

@alanwest alanwest Feb 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sanjalee-NewRelic please rename the file to be specific to AWS EXT-SERVICE-to-INFRA-AWSMQBROKER.yml

Original file line number Diff line number Diff line change
@@ -1,16 +1,87 @@
relationships:
- name: extServiceConsumesAwsMqBroker
- name: extServiceConsumesAwsMqBroker1_20
version: "1"
origins:
- Distributed Tracing
- OpenTelemetry
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "consumer" ]
anyOf: [ "consumer", "server" ]
- attribute: messaging.system
anyOf: [ "rabbitmq", "activemq" ]
- attribute: net.peer.name
present: true
regex: "^[^\\.]+\\.mq\\.[^\\.]+\\.amazonaws\\.com"
relationship:
expires: P75M
relationshipType: CONSUMES
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: AWSMQBROKER
fields:
- field: endpoint
attribute: net.peer.name
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, this lookup onMiss will create an an uninstrumented MQBROKER entity.

entity-definitions/relationships/candidates/AWSMQBROKER.yml

Lines 13 to 16 in dbf47cc

onMiss:
action: CREATE_UNINSTRUMENTED
uninstrumented:
type: MQBROKER

In the future if we add new relationships supporting messaging systems besides AWS (e.g., RabbitMQ hosted elsewhere), will this rule cause us to falsely create uninstrumented relationships?

If so, then we may need to constrain this rule to be AWS specific. Maybe like so?

    conditions:
      - attribute: net.peer.name
         regex: <regex that identifies this is AWS>


- name: extServiceProducesAwsMqBroker1_20
version: "1"
origins:
- Distributed Tracing
- OpenTelemetry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not entirely clear what origins should be used here. In reviewing other rules it seems we are inconsistent.

For example, the infra-host to ext-service rule derived from span data only uses an origin of Distributed Tracing:

entity-definitions/relationships/synthesis/INFRA-HOST-to-EXT-SERVICE.yml

Lines 4 to 5 in dbf47cc

origins:
- Distributed Tracing

Whereas the the infra-host to ext-service rule derived from metric data only uses an origin of OpenTelemetry:

entity-definitions/relationships/synthesis/INFRA-HOST-to-EXT-SERVICE.yml

Lines 37 to 38 in dbf47cc

origins:
- OpenTelemetry

I can see the following origins being relevant to the rules in this PR:

  • Distributed Tracing because these rules target span data.
  • OpenTelemetry because these rules are meant to target OpenTelemetry instrumented services
  • AWS Integration because these rules target our AWS Metric Streams integration

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@otaviocarvalho can you provide some guidance here? From what I can tell these origins have been inconsistently applied across other relationship rules, so it doesn't seem like they're very important.

All three origins are relevant to the domain of these rules. Is the practice that we should list all relevant origins? Or does it not really matter?

conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "producer", "client" ]
- attribute: messaging.system
anyOf: [ "rabbitmq", "activemq" ]
- attribute: net.peer.name
present: true
regex: "^[^\\.]+\\.mq\\.[^\\.]+\\.amazonaws\\.com"
relationship:
expires: P75M
relationshipType: PRODUCES
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: AWSMQBROKER
fields:
- field: endpoint
attribute: net.peer.name

- name: extServiceConsumesAwsMqBroker1_23
version: "1"
origins:
- Distributed Tracing
- OpenTelemetry
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "consumer", "server" ]
- attribute: messaging.system
anyOf: [ "rabbitmq", "activemq" ]
- attribute: server.address
present: true
regex: "^[^\\.]+\\.mq\\.[^\\.]+\\.amazonaws\\.com"
relationship:
expires: P75M
relationshipType: CONSUMES
Expand All @@ -24,18 +95,25 @@ relationships:
- field: endpoint
attribute: server.address

- name: extServiceProducesAwsMqBroker
- name: extServiceProducesAwsMqBroker1_23
version: "1"
origins:
- Distributed Tracing
- OpenTelemetry
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "producer" ]
anyOf: [ "producer", "client" ]
- attribute: messaging.system
anyOf: [ "rabbitmq", "activemq" ]
- attribute: server.address
present: true
regex: "^[^\\.]+\\.mq\\.[^\\.]+\\.amazonaws\\.com"
relationship:
expires: P75M
relationshipType: PRODUCES
Expand Down
107 changes: 102 additions & 5 deletions relationships/synthesis/EXT-SERVICE-to-KAFKATOPIC.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,23 @@
relationships:
- name: extServiceProducesAwsMskTopic
- name: extServiceProducesAwsMskTopic1_20
version: "1"
origins:
- Distributed Tracing
- OpenTelemetry
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "producer" ]
anyOf: [ "producer", "client" ]
- attribute: net.peer.name
present: true
regex: "^[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
- attribute: messaging.destination.name
present: true
relationship:
expires: P75M
relationshipType: PRODUCES
Expand All @@ -20,7 +30,85 @@ relationships:
fields:
- field: clusterName
capture:
attribute: server.address
attribute: net.peer.name
regex: "^[^\\.]+\\.([^\\.]+)\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
Comment on lines +33 to +34
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to my question above, should this rule be constrained by adding a condition that makes it specific to AWS?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably being too pragmatic, but if the domain includes amazonaws...com there is no way it will not be specific to AWS. Alternatively, you can add and check a specific tag for this purpose.

- field: region
capture:
attribute: net.peer.name
regex: "^[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.kafka\\.([^\\.]+)\\.amazonaws\\.com:[0-9]+.*"
- field: topic
attribute: messaging.destination.name

- name: extServiceConsumesAwsMskTopic1_20
version: "1"
origins:
- Distributed Tracing
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "consumer", "server" ]
- attribute: net.peer.name
present: true
regex: "^[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
- attribute: messaging.destination.name
present: true
relationship:
expires: P75M
relationshipType: CONSUMES
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: KAFKATOPIC
fields:
- field: clusterName
capture:
attribute: net.peer.name
regex: "^[^\\.]+\\.([^\\.]+)\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
- field: region
capture:
attribute: net.peer.name
regex: "^[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.kafka\\.([^\\.]+)\\.amazonaws\\.com:[0-9]+.*"
- field: topic
attribute: messaging.destination.name
Comment on lines +78 to +79
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

messaging.destination.name is not guaranteed to exist. If these rules require it be present then it should be added to the conditions

    conditions:
      ...
      - attribute: messaging.destination.name
         present: true


- name: extServiceProducesAwsMskTopic1_23
version: "1"
origins:
- Distributed Tracing
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "producer", "client" ]
- attribute: server.address
present: true
regex: "^[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
- attribute: messaging.destination.name
present: true
relationship:
expires: P75M
relationshipType: PRODUCES
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: KAFKATOPIC
fields:
- field: clusterName
capture:
attribute: server.address
regex: "^[^\\.]+\\.([^\\.]+)\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
- field: region
capture:
Expand All @@ -29,15 +117,24 @@ relationships:
- field: topic
attribute: messaging.destination.name

- name: extServiceConsumesAwsMskTopic
- name: extServiceConsumesAwsMskTopic1_23
version: "1"
origins:
- Distributed Tracing
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: span.kind
anyOf: [ "consumer" ]
anyOf: [ "consumer", "server" ]
- attribute: server.address
present: true
regex: "^[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.[^\\.]+\\.kafka\\.[^\\.]+\\.amazonaws\\.com:[0-9]+.*"
- attribute: messaging.destination.name
present: true
relationship:
expires: P75M
relationshipType: CONSUMES
Expand Down
111 changes: 108 additions & 3 deletions relationships/synthesis/EXT-SERVICE-to-MEMCACHED.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,86 @@
relationships:
- name: extServiceCallsMemcached
- name: extServiceCallsMemcached1_20
version: "1"
origins:
- Distributed Tracing
- OpenTelemetry
conditions:
- attribute: eventType
anyOf: [ "Span"]
- attribute: db.system.name
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: db.system
anyOf: [ "memcached" ]
- attribute: net.peer.name
present: true
- attribute: net.peer.port
present: true
relationship:
expires: P75M
relationshipType: CALLS
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: MEMCACHED
fields:
- field: endpoint
attribute: net.peer.name
- field: port
attribute: net.peer.port

- name: extServiceCallsMemcached1_20_Default_Port
version: "1"
origins:
- Distributed Tracing
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: db.system
anyOf: [ "memcached" ]
- attribute: net.peer.name
present: true
- attribute: net.peer.port
present: false
relationship:
expires: P75M
relationshipType: CALLS
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: MEMCACHED
fields:
- field: endpoint
attribute: net.peer.name
- field: port
attribute: 11211 # Default port for memcached

- name: extServiceCallsMemcached1_23
version: "1"
origins:
- Distributed Tracing
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: db.system
anyOf: [ "memcached" ]
- attribute: server.address
present: true
- attribute: server.port
present: true
relationship:
expires: P75M
relationshipType: CALLS
Expand All @@ -22,3 +95,35 @@ relationships:
attribute: server.address
- field: port
attribute: server.port

- name: extServiceCallsMemcached1_23_Default_Port
version: "1"
origins:
- Distributed Tracing
conditions:
- attribute: eventType
anyOf: [ "Span" ]
- attribute: newrelic.source
anyOf: [ "api.traces.otlp" ]
- attribute: entity.type
anyOf: [ "SERVICE" ]
- attribute: db.system
anyOf: [ "memcached" ]
Comment on lines +113 to +114
Copy link
Member

@alanwest alanwest Feb 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sanjalee-NewRelic After reviewing the rules for redis, it looks like they are restricted to AWS.

See: https://github.com/newrelic/entity-definitions/pull/1904/files#diff-f8b949e2507623f79e34ed7186ae981c06082c0e27013bce60088e0d0de9beb2R52

I think it makes sense to do the same for memcached, right?

Something like

- attribute: server.address
   regex: <regex constraining this to AWS>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, This can be done. Made the change.

- attribute: server.address
present: true
- attribute: server.port
present: false
relationship:
expires: P75M
relationshipType: CALLS
source:
extractGuid:
attribute: entity.guid
target:
lookupGuid:
candidateCategory: MEMCACHED
fields:
- field: endpoint
attribute: server.address
- field: port
attribute: 11211 # Default port for memcached