Adding AgentHealth Middleware to Exporters

[Paramadon]

---

Description of the Issue

The CloudWatch Agent lacked an agenthealth configuration to monitor API health by tracking HTTP status codes for specific responses. Monitoring the status codes (200, 400, 408, 419, and 429) across all APIs is critical for diagnosing issues and ensuring comprehensive observability of API behaviors. Without this configuration, users were unable to quickly identify trends in API health metrics or correlate specific status codes with performance issues.

---

Changes Made

This change adds agent health to the emf_exporter and awscloudwatchlogs exporter. AwsXrayExporter already has handlers being added.

• Added agenthealth Configuration:

  • Configured the CloudWatch Agent to track the following status codes across all APIs:
    • 200: Success
    • 400: Bad Request
    • 408: Request Timeout
    • 419: Authentication Timeout
    • 429: Too Many Requests
  • This ensures that health metrics for these common response codes are captured and reported.

• Updated CloudWatch Agent Configuration JSON:

  • Introduced a new section under the metrics collection configuration to specify agenthealth metrics.
  • Ensured compatibility with existing metrics and logs collection by integrating the new configuration seamlessly.

• Validated the Configuration:

  • Tested the updated configuration with the CloudWatch Agent to ensure the new metrics are collected and reported correctly.
  • Confirmed that the metrics appear as expected in CloudWatch for all specified APIs.

---

Impact

• Provides real-time visibility into API status codes, helping identify anomalies or patterns that indicate issues.
• Enhances observability for critical services by adding detailed health metrics.

---

Testing

• Deployed the updated CloudWatch Agent health configuraion
• Verified that metrics for status codes 200, 400, 408, 419, and 429 are captured in the header as you can see from the image below:

---

[dricross]

Why change from uuid.NewRandom()?

[Paramadon]

Changed it back to uuid.NewRandom().

[dricross]

good find

[dricross]

I'm not super familiar with this exporter, but is it possible that that the reference to xrayClient on line 78 could happen before the initialization in the exporthelper.WithStart?

[Paramadon]

So based off the BaseExporter struct and the Start funciton of the Base exporter, the Start function is called before the exporter begins processing data. I believe this is a standard practice in the OpenTelemetry Collector's component lifecycle.

// baseExporter contains common fields between different exporter types.
type baseExporter struct {
	component.StartFunc
	component.ShutdownFunc

	signal component.DataType

	batchMergeFunc      exporterbatcher.BatchMergeFunc[Request]
	batchMergeSplitfunc exporterbatcher.BatchMergeSplitFunc[Request]

	marshaler   exporterqueue.Marshaler[Request]
	unmarshaler exporterqueue.Unmarshaler[Request]

	set    exporter.Settings
	obsrep *ObsReport

	// Message for the user to be added with an export failure message.
	exportFailureMessage string

	// Chain of senders that the exporter helper applies before passing the data to the actual exporter.
	// The data is handled by each sender in the respective order starting from the queueSender.
	// Most of the senders are optional, and initialized with a no-op path-through sender.
	batchSender   requestSender
	queueSender   requestSender
	obsrepSender  requestSender
	retrySender   requestSender
	timeoutSender *timeoutSender // timeoutSender is always initialized.

	consumerOptions []consumer.Option
}

func (be *baseExporter) Start(ctx context.Context, host component.Host) error {
	// First start the wrapped exporter.
	if err := be.StartFunc.Start(ctx, host); err != nil {
		return err
	}

	// If no error then start the batchSender.
	if err := be.batchSender.Start(ctx, host); err != nil {
		return err
	}

	// Last start the queueSender.
	return be.queueSender.Start(ctx, host)
}

[okankoAMZ]

can we name this something like exporter settings?

[okankoAMZ]

can we name this settings

[okankoAMZ]

non-blocking comments