Skip to content

Commit

Permalink
[azureeventhubreceiver] port PR
Browse files Browse the repository at this point in the history
  • Loading branch information
nslaughter committed Jun 6, 2024
1 parent 045455c commit 97bee5d
Show file tree
Hide file tree
Showing 25 changed files with 2,304 additions and 0 deletions.
1 change: 1 addition & 0 deletions collector/components/azureeventhubreceiver/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
125 changes: 125 additions & 0 deletions collector/components/azureeventhubreceiver/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Azure Event Hub Receiver

<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [alpha]: metrics, logs |
| Distributions | [contrib], [observiq], [splunk], [sumo] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fazureeventhub%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fazureeventhub) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fazureeventhub%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fazureeventhub) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme), [@djaglowski](https://www.github.com/djaglowski) |

[alpha]: https://github.com/open-telemetry/opentelemetry-collector#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[observiq]: https://github.com/observIQ/observiq-otel-collector
[splunk]: https://github.com/signalfx/splunk-otel-collector
[sumo]: https://github.com/SumoLogic/sumologic-otel-collector
<!-- end autogenerated section -->

## Overview
Azure resources and services can be
[configured](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings)
to send their logs to an Azure Event Hub. The Azure Event Hub receiver pulls logs from an Azure
Event Hub, transforms them, and pushes them through the collector pipeline.

## Configuration

### connection (Required)
A string describing the connection to an Azure event hub.

### group (Optional)
The Consumer Group to read from. If empty will default to the default Consumer Group $Default

### partition (Optional)
The partition to watch. If empty, it will watch explicitly all partitions.

Default: ""

### offset (Optional)
The offset at which to start watching the event hub. If empty, it starts with the latest offset.

Default: ""

### format (Optional)
Determines how to transform the Event Hub messages into OpenTelemetry logs. See the "Format"
section below for details.

Default: "azure"

### Example Configuration

```yaml
receivers:
azureeventhub:
connection: Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=superSecret1234=;EntityPath=hubName
partition: foo
group: bar
offset: "1234-5566"
format: "azure"
```
This component can persist its state using the [storage extension].
## Format
### raw
The "raw" format maps the AMQP properties and data into the
attributes and body of an OpenTelemetry LogRecord, respectively.
The body is represented as a raw byte array.
This format is not supported for Metrics.
### azure
The "azure" format extracts the Azure log records from the AMQP
message data, parses them, and maps the fields to OpenTelemetry
attributes. The table below summarizes the mapping between the
[Azure common log format](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/resource-logs-schema)
and the OpenTelemetry attributes.
| Azure | OpenTelemetry |
|----------------------------------|----------------------------------------|
| callerIpAddress (optional) | net.sock.peer.addr (attribute) |
| correlationId (optional) | azure.correlation.id (attribute) |
| category (optional) | azure.category (attribute) |
| durationMs (optional) | azure.duration (attribute) |
| Level (optional) | severity_number, severity_text (field) |
| location (optional) | cloud.region (attribute) |
| — | cloud.provider (attribute) |
| operationName (required) | azure.operation.name (attribute) |
| operationVersion (optional) | azure.operation.version (attribute) |
| properties (optional) | azure.properties (attribute, nested) |
| resourceId (required) | azure.resource.id (resource attribute) |
| resultDescription (optional) | azure.result.description (attribute) |
| resultSignature (optional) | azure.result.signature (attribute) |
| resultType (optional) | azure.result.type (attribute) |
| tenantId (required, tenant logs) | azure.tenant.id (attribute) |
| time or timeStamp (required) | time_unix_nano (time takes precedence) |
| identity (optional) | azure.identity (attribute, nested) |
Notes:
* JSON does not distinguish between fixed and floating point numbers. All
JSON numbers are encoded as doubles.
For Metrics the Azure Metric Records are an array
of "records" with the following fields.
| Azure | Open Telemetry |
|------------|---------------------------------------------|
| time | time_unix_nano (field) |
| resourceId | azure.resource.id (resource attribute) |
| metricName | |
| timeGrain | start_time_unix_nano (field) |
| total | mapped to datapoint metricName + "_TOTAL" |
| count | mapped to datapoint metricName + "_COUNT" |
| minimum | mapped to datapoint metricName + "_MINIMUM" |
| maximum | mapped to datapoint metricName + "_MAXIMUM" |
| average | mapped to datapoint metricName + "_AVERAGE" |
From this data a Metric of type Gauge is created
with a Data Points that represents the values
for the Metric including: Total, Minimum, Maximum,
Average and Count.
[storage extension]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package azureeventhubreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/azureeventhubreceiver"

import (
"github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs"
"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/pdata/plog"
"go.uber.org/zap"

"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/azure"
)

type AzureResourceLogsEventUnmarshaler struct {
unmarshaler *azure.ResourceLogsUnmarshaler
}

func newAzureResourceLogsUnmarshaler(buildInfo component.BuildInfo, logger *zap.Logger) eventLogsUnmarshaler {
return AzureResourceLogsEventUnmarshaler{
unmarshaler: &azure.ResourceLogsUnmarshaler{
Version: buildInfo.Version,
Logger: logger,
},
}
}

// UnmarshalLogs takes a byte array containing a JSON-encoded
// payload with Azure log records and transforms it into
// an OpenTelemetry plog.Logs object. The data in the Azure
// log record appears as fields and attributes in the
// OpenTelemetry representation; the bodies of the
// OpenTelemetry log records are empty.
func (r AzureResourceLogsEventUnmarshaler) UnmarshalLogs(event *azeventhubs.ReceivedEventData) (plog.Logs, error) {
return r.unmarshaler.UnmarshalLogs(event.Body)
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 97bee5d

Please sign in to comment.