From 55bd345c5a8873def851bf6fa89dd7756c31509e Mon Sep 17 00:00:00 2001 From: Peter Souter Date: Thu, 25 Sep 2025 18:30:28 +0100 Subject: [PATCH 1/3] docs: add Apache trace-to-log correlation configuration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add section on correlating traces to Apache logs - Include LogFormat directive with trace/span ID injection - Provide log pipeline configuration steps for ID parsing - Add required reference links for log processors 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- .../trace_collection/proxy_setup/httpd.md | 66 +++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/content/en/tracing/trace_collection/proxy_setup/httpd.md b/content/en/tracing/trace_collection/proxy_setup/httpd.md index d6ffb9dde0ca7..5b4c0152761f1 100644 --- a/content/en/tracing/trace_collection/proxy_setup/httpd.md +++ b/content/en/tracing/trace_collection/proxy_setup/httpd.md @@ -74,6 +74,68 @@ DatadogServiceName my-app DatadogSamplingRate 0.1 ``` +## Correlating traces to logs + +After you've enabled APM tracing, you can connect your traces to the corresponding Apache logs. This correlation links each trace and span to the specific log events generated during that request, allowing you to pivot between them to troubleshoot issues. + +### Prerequisites + +Before you begin, ensure that you have: +- Enabled APM tracing for Apache HTTP Server by following the steps earlier in this guide. +- Configured Datadog log collection for Apache HTTP Server. + +### Step 1: Inject trace and span IDs into Apache logs + +Modify your `LogFormat` directive to include the trace ID and the span ID using the Apache-specific variables `%{Datadog-Trace-ID}e` and `%{Datadog-Span-ID}e`. These values are `-` for requests that are not traced. + +Update your Apache configuration file (for example, `/etc/apache2/apache2.conf` or `/etc/httpd/httpd.conf`): + +```apache +LogFormat "%h %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\" \"%{X-Forwarded-For}i\" dd.trace_id=\"%{Datadog-Trace-ID}e\" dd.span_id=\"%{Datadog-Span-ID}e\"" datadog_combined + +CustomLog /var/log/apache2/access.log datadog_combined +``` + +After saving your changes, reload the Apache configuration. For example: + +```sh +sudo systemctl reload apache2 +# or +sudo apachectl -k graceful +``` + +### Step 2: Configure the log pipeline to parse the trace and span IDs + +By default, logs capture `dd.trace_id` and `dd.span_id` in hexadecimal format: + +``` +172.237.96.114 - - [25/Sep/2025:14:48:52 +0000] "GET / HTTP/1.1" 200 617 28248 "-" "curl/8.7.1" "-" dd.trace_id="68d5565400000000970bf1a70782d612" dd.span_id="10884058624158782994" +``` + +To enable log and trace correlation in Datadog, configure a log processing pipeline to convert these IDs from hexadecimal to decimal. + +1. In Datadog, navigate to the [**Log Configuration**][9] page. +2. Hover over your active Apache pipeline and click the **Clone** icon to create an editable version. +3. Click the cloned pipeline. +4. Click **Add Processor**. +5. Select [Grok Parser][11] as the processor type. +6. Define the following parsing rule to extract the trace ID attribute from a log event: + ```text + extract_correlation_ids %{data} dd.trace_id="%{notSpace:dd.trace_id:nullIf("-")}" dd.span_id="%{notSpace:dd.span_id:nullIf("-")}" + ``` +7. Click **Create**. +8. Click **Add Processor** again. +9. Select [Trace ID Remapper][10] as the processor type. This processor associates the parsed ID with its corresponding APM trace. +10. In the **Set trace id attribute(s)** field, enter `dd.trace_id`. +11. Click **Create**. +12. Click **Add Processor** again. +13. Select [Span ID Remapper][12] as the processor type. This processor associates the parsed ID with its corresponding APM span. +14. In the **Set span id attribute(s)** field, enter `dd.span_id`. +15. Click **Create**. +16. Save and enable your new pipeline. + +Once the pipeline is active, new Apache logs are automatically correlated with their traces and spans. + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} @@ -81,3 +143,7 @@ DatadogSamplingRate 0.1 [1]: https://github.com/DataDog/httpd-datadog [2]: https://httpd.apache.org/ [3]: https://github.com/DataDog/httpd-datadog/blob/main/doc/configuration.md +[9]: https://app.datadoghq.com/logs/pipelines +[10]: /logs/log_configuration/processors/?tab=ui#trace-remapper +[11]: /logs/log_configuration/processors/?tab=ui#grok-parser +[12]: /logs/log_configuration/processors/?tab=ui#span-remapper From 6b56c14b56032ec63443f2d3b8c19e08c02af6cb Mon Sep 17 00:00:00 2001 From: Peter Souter Date: Fri, 26 Sep 2025 00:30:35 +0100 Subject: [PATCH 2/3] docs: slight working tweak Co-authored-by: Rosa Trieu <107086888+rtrieu@users.noreply.github.com> --- content/en/tracing/trace_collection/proxy_setup/httpd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/tracing/trace_collection/proxy_setup/httpd.md b/content/en/tracing/trace_collection/proxy_setup/httpd.md index 5b4c0152761f1..49d42a2cf9da8 100644 --- a/content/en/tracing/trace_collection/proxy_setup/httpd.md +++ b/content/en/tracing/trace_collection/proxy_setup/httpd.md @@ -134,7 +134,7 @@ To enable log and trace correlation in Datadog, configure a log processing pipel 15. Click **Create**. 16. Save and enable your new pipeline. -Once the pipeline is active, new Apache logs are automatically correlated with their traces and spans. +After the pipeline is active, new Apache logs are automatically correlated with their traces and spans. ## Further Reading From 5272d196466d0b80ea6755b8f594bbd963d9aa3b Mon Sep 17 00:00:00 2001 From: Peter Souter Date: Thu, 25 Sep 2025 18:30:28 +0100 Subject: [PATCH 3/3] docs: add Apache trace-to-log correlation configuration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add section on correlating traces to Apache logs - Include LogFormat directive with trace/span ID injection - Provide log pipeline configuration steps for ID parsing - Add required reference links for log processors 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- .../trace_collection/proxy_setup/httpd.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/content/en/tracing/trace_collection/proxy_setup/httpd.md b/content/en/tracing/trace_collection/proxy_setup/httpd.md index 49d42a2cf9da8..5a11dd2e96292 100644 --- a/content/en/tracing/trace_collection/proxy_setup/httpd.md +++ b/content/en/tracing/trace_collection/proxy_setup/httpd.md @@ -114,27 +114,27 @@ By default, logs capture `dd.trace_id` and `dd.span_id` in hexadecimal format: To enable log and trace correlation in Datadog, configure a log processing pipeline to convert these IDs from hexadecimal to decimal. -1. In Datadog, navigate to the [**Log Configuration**][9] page. +1. In Datadog, navigate to the [**Log Configuration**][4] page. 2. Hover over your active Apache pipeline and click the **Clone** icon to create an editable version. 3. Click the cloned pipeline. 4. Click **Add Processor**. -5. Select [Grok Parser][11] as the processor type. +5. Select [Grok Parser][5] as the processor type. 6. Define the following parsing rule to extract the trace ID attribute from a log event: ```text extract_correlation_ids %{data} dd.trace_id="%{notSpace:dd.trace_id:nullIf("-")}" dd.span_id="%{notSpace:dd.span_id:nullIf("-")}" ``` 7. Click **Create**. 8. Click **Add Processor** again. -9. Select [Trace ID Remapper][10] as the processor type. This processor associates the parsed ID with its corresponding APM trace. +9. Select [Trace ID Remapper][6] as the processor type. This processor associates the parsed ID with its corresponding APM trace. 10. In the **Set trace id attribute(s)** field, enter `dd.trace_id`. 11. Click **Create**. 12. Click **Add Processor** again. -13. Select [Span ID Remapper][12] as the processor type. This processor associates the parsed ID with its corresponding APM span. +13. Select [Span ID Remapper][7] as the processor type. This processor associates the parsed ID with its corresponding APM span. 14. In the **Set span id attribute(s)** field, enter `dd.span_id`. 15. Click **Create**. 16. Save and enable your new pipeline. -After the pipeline is active, new Apache logs are automatically correlated with their traces and spans. +Once the pipeline is active, new Apache logs are automatically correlated with their traces and spans. ## Further Reading @@ -143,7 +143,7 @@ After the pipeline is active, new Apache logs are automatically correlated with [1]: https://github.com/DataDog/httpd-datadog [2]: https://httpd.apache.org/ [3]: https://github.com/DataDog/httpd-datadog/blob/main/doc/configuration.md -[9]: https://app.datadoghq.com/logs/pipelines -[10]: /logs/log_configuration/processors/?tab=ui#trace-remapper -[11]: /logs/log_configuration/processors/?tab=ui#grok-parser -[12]: /logs/log_configuration/processors/?tab=ui#span-remapper +[4]: https://app.datadoghq.com/logs/pipelines +[5]: /logs/log_configuration/processors/?tab=ui#grok-parser +[6]: /logs/log_configuration/processors/?tab=ui#trace-remapper +[7]: /logs/log_configuration/processors/?tab=ui#span-remapper