Java Agent ignoring environment variables and java parameters for service name

[decimalst]

Describe the bug

Hi folks, I am trying to instrument my service with the Otel java agent. For some reason however, the agent appears to be ignoring specific parameters or environment variables, such as OTEL_SERVICE_NAME.

Steps to reproduce

This happens when invoking the app with

OTEL_SERVICE_NAME=validator_service java -javaagent:/path/to/otel-2.10.0.jar -DapplicationName=ValidatorService -Ddebug=true -Dlogback.debug=true -DworkingDir=/path/to/app/linux64/version -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:+UseG1GC -XX:+PrintGCDetails -Xloggc:log/gc-2024-11-26.14.41.23.log -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -Dotel.javaagent.enabled=true -Dotel.traces.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider -Dotel.java.disabled.resource.providers=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider,io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider -Dotel.javaagent.logging=application -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true -Dotel.instrumentation.jetty.enabled=true -Dotel.instrumentation.jetty-httpclient.enabled=true -Dotel.instrumentation.vertx-http-client.enabled=true -Dotel.instrumentation.vertx-redis-client.enabled=false -Dotel.instrumentation.vertx-rx-java.enabled=true -Dotel.instrumentation.vertx-sql-client.enabled=true -Dotel.instrumentation.vertx-web.enabled=true -Dotel.instrumentation.jdbc.enabled=true -Dotel.instrumentation.http-url-connection.enabled=true -Dotel.instrumentation.jms.enabled=true -Dotel.instrumentation.apache-httpclient.enabled=false -classpath [class path libraries] path.to.my.ValidationService

Expected behavior

The otel service name to be validator_service in Datadog, and traces being produced

Actual behavior

Traces are produced, but only for unknown_service:java in Datadog. I can see the configuration parameters reflected in the telemetry as process args.

When I enable debug logs, I see the log line:

..., process.pid=, service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}

So it seems the autoconfigured SDK builder is being created and picking up some attributes, but not all

Javaagent or library instrumentation version

2.10.0

Environment

JDK: jdk1.8.0_173
OS: RHEL7

Additional context

No response

[jaydeluca]

I am able to observe the agent picking up the environment variable OTEL_SERVICE_NAME as expected when running an application in the format you provided.

I don't see you passing a javaagent in your command, how are you using the instrumentation?

[decimalst]

Hey @jaydeluca - Thanks for your response. I accidentally stripped the -javaagent:path/to/jar command because it had a username in the path, but edited it for clarity now. I did double check that the -javaagent param is the first param after the java command.
Here's another sample log line: I am trying to provide configuration via the arg -Dotel.javaagent.configuration.file=/home/username/code/servicename/otel.properties

Inside the otel.properties files I have otel.service.name defined:

# OpenTelemetry Java Agent Configuration

# Core Settings
otel.javaagent.enabled=true
otel.service.name=test_servicename

# Exporters Configuration
otel.traces.exporter=otlp
otel.logs.exporter=otlp
otel.metrics.exporter=otlp
otel.exporter.otlp.endpoint=http://localhost:4318

In the app logs

figuredOpenTelemetrySdkBuilder Global OpenTelemetry set to OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="hostname", process.command_line="/path/to/my/jdk1.8.0_173/jre/bin/java -javaagent:/home/username/otel.jar -DapplicationName=Validator -Dotel.javaagent.configuration.file=/home/username/code/servicename/otel.properties
...
service.name="unknown_service:java"

Or, if I try -Dotel.service.name=some-service-name

2024-11-29T13:59:21.630 DEBUG [main                          ] figuredOpenTelemetrySdkBuilder Global OpenTelemetry set to OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="ch12ldvdi457", process.command_line="/java/linux64/jdk1.8.0_173/jre/bin/java -javaagent:/home/username/otel.jar -Dotel.service.name=my_service_name -DapplicationName=FitValidator  -Ddebug=true -Dlogback.debug=true -DworkingDir=/app/linux64/someversion -Dcom.sun.management.jmxremote.ssl=false -Dotel.javaagent.enabled=true -Dotel.traces.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider -Dotel.java.disabled.resource.providers=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider,io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider -Dotel.javaagent.logging=application -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true -Dotel.instrumentation.jetty.enabled=true -Dotel.instrumentation.jetty-httpclient.enabled=true -Dotel.instrumentation.vertx-http-client.enabled=true -Dotel.instrumentation.vertx-redis-client.enabled=false -Dotel.instrumentation.vertx-rx-java.enabled=true -Dotel.instrumentation.vertx-sql-client.enabled=true -Dotel.instrumentation.vertx-web.enabled=true -Dotel.instrumentation.jdbc.enabled=true -Dotel.instrumentation.http-url-connection.enabled=true -Dotel.instrumentation.jms.enabled=true -Dotel.instrumentation.apache-httpclient.enabled=false com.chicagotrading.fod.fitting.validation.ValidationService",  service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}, s, process.pid=1646, service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}, metricReaders=[PeriodicMetricReader{exporter=OtlpHttpMetricExporter{exporterName=otlp, type=metric, endpoint=http://localhost:4318/v1/metrics, timeoutNanos=10000000000, proxyOptions=null, compressorEncoding=null, connectTimeoutNanos=10000000000, exportAsJson=false, headers=Headers{User-Agent=OBFUSCATED}, retryPolicy=RetryPolicy{maxAttempts=5, initialBackoff=PT1S, maxBackoff=PT5S, backoffMultiplier=1.5}, aggregationTemporalitySelector=AggregationTemporalitySelector{COUNTER=CUMULATIVE, UP_DOWN_COUNTER=CUMULATIVE, HISTOGRAM=CUMULATIVE, OBSERVABLE_COUNTER=CUMULATIVE, OBSERVABLE_UP_DOWN_COUNTER=CUMULATIVE, OBSERVABLE_GAUGE=CUMULATIVE, GAUGE=CUMULATIVE}, defaultAggregationSelector=DefaultAggregationSelector{COUNTER=default, UP_DOWN_COUNTER=default, HISTOGRAM=default, OBSERVABLE_COUNTER=default, OBSERVABLE_UP_DOWN_COUNTER=default, OBSERVABLE_GAUGE=default, GAUGE=default}, memoryMode=REUSABLE_DATA}, intervalNanos=60000000000}], metricProducers=[], views=[]}, loggerProvider=SdkLoggerProvider{clock=SystemClock{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="", process.command_line="/java/linux64/jdk1.8.0_173/jre/bin/java -javaagent:/home/username/otel.jar -Dotel.service.name=my_service_name -DapplicationName=Validator -Ddebug=true -Dlogback.debug=true -DworkingDir=/app/linux64/someversion -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -XX:+UseG1GC -XX:+PrintGCDetails -Xloggc:log/gc-2024-11-26.14.41.23.log -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -Dotel.javaagent.enabled=true -Dotel.traces.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider -Dotel.java.disabled.resource.providers=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider,io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider -Dotel.javaagent.logging=application -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.opentelemetry-api.enabled=true -Dotel.instrumentation.opentelemetry-instrumentation-annotations.enabled=true -Dotel.instrumentation.jetty.enabled=true -Dotel.instrumentation.jetty-httpclient.enabled=true -Dotel.instrumentation.vertx-http-client.enabled=true -Dotel.instrumentation.vertx-redis-client.enabled=false -Dotel.instrumentation.vertx-rx-java.enabled=true -Dotel.instrumentation.vertx-sql-client.enabled=true -Dotel.instrumentation.vertx-web.enabled=true -Dotel.instrumentation.jdbc.enabled=true -Dotel.instrumentation.http-url-connection.enabled=true -Dotel.instrumentation.jms.enabled=true -Dotel.instrumentation.apache-httpclient.enabled=false com.chicagotrading.fod.fitting.validation.ValidationService", process.executable.path="/java/linux64/jdk1.8.0_173/jre/bin/java", process.pid=1646, service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.44.1"}}, logLimits=LogLimits{maxNumberOfAttributes=128, maxAttributeValueLength=2147483647}, logRecordProcessor=BatchLogRecordProcessor{logRecordExporter=OtlpHttpLogRecordExporter{exporterName=otlp, type=log, endpoint=http://localhost:4318/v1/logs, timeoutNanos=10000000000, proxyOptions=null, compressorEncoding=null, connectTimeoutNanos=10000000000, exportAsJson=false, headers=Headers{User-Agent=OBFUSCATED}, retryPolicy=RetryPolicy{maxAttempts=5, initialBackoff=PT1S, maxBackoff=PT5S, backoffMultiplier=1.5}, memoryMode=REUSABLE_DATA}, scheduleDelayNanos=1000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}}, propagators=DefaultContextPropagators{textMapPropagator=MultiTextMapPropagator{textMapPropagators=[W3CTraceContextPropagator, W3CBaggagePropagator]}}} by autoconfiguration

I see that the process arg is there, and even being capture as a process arg, but I still see service.name="unknown_service:java"

[decimalst]

It feels like this issue may be specific to my environment due to our jdk version?
E.g. I do see in some of the debug logs:

2024-11-29T13:59:21.629 DEBUG [main                          ] i.o.j.t.SafeServiceLoader      Unable to load instrumentation class: io/opentelemetry/javaagent/shaded/instrumentation/javaagent/runtimemetrics/java17/Java17RuntimeMetricsInstaller has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 52.0

But I assumed I would see that same error for the autoconfiguration class if that's why it wasn't catching specific parameters or env vars e.g.

[jaydeluca]

the property for the javaagent config file should be -Dotel.javaagent.configuration-file instead of -Dotel.javaagent.configuration.file

You could also try using the new declarative-config approach

[decimalst]

Ah, thanks - tried again with -Dotel.javaagent.configuration-file=/home/username/code/service/otel.properties but no change in behavior, it still seems to be ignoring the otel.service.name specified in the properties file.

[jaydeluca]

would you be able to provide a link to a github repo with a minimal app that reproduces the issue? I'm able to use the same properties file you provided and I see the service name as expected

[laurit]

service.name is set by a resource provider https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/internal/EnvironmentResourceProvider.java. Using -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.HostResourceProvider,io.opentelemetry.instrumentation.resources.ProcessResourceProvider disables all other resource providers except what you have listed, including the provider that sets the service.name. cc @jack-berg

[decimalst]

@laurit Ah - thank you, that appears to have been the fix, as after removing that line I do see service names in my traces!
So basically I would need to add -Dotel.java.enabled.resource.providers=**io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider**,others ?
Edit: Is it worth adding that to the docs here https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/languages/java/configuration.md ? Happy to raise a PR to do so

[jack-berg]

So basically I would need to add -Dotel.java.enabled.resource.providers=io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider,others ?

Edit: Is it worth adding that to the docs here https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/languages/java/configuration.md ? Happy to raise a PR to do so

There is some description of otel.java.enabled.resource.providers here. Perhaps it could use some clarification, but I'm curious what your interpretation was.

Going back the config from the original issue, you had (formatted as a list for readability):

• Dotel.java.enabled.resource.providers
  • io.opentelemetry.instrumentation.resources.HostResourceProvider
  • io.opentelemetry.instrumentation.resources.ProcessResourceProvider
• Dotel.java.disabled.resource.providers
  • io.opentelemetry.contrib.gcp.resource.GCPResourceProvider
  • io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider

The way this worked:

• Setting otel.java.enabled.resource.providers causes only specified HostResourceProvider and ProcessResourceProvider to be active
• Setting otel.java.disabled.resource.providers overrides otel.java.enabled.resource.providers, but does nothing in this case because neither GCPResourceProvider or Ec2ResourceProvider are enabled

[decimalst]

Going back the config from the original issue, you had (formatted as a list for readability):

• Dotel.java.enabled.resource.providers

  • io.opentelemetry.instrumentation.resources.HostResourceProvider
  • io.opentelemetry.instrumentation.resources.ProcessResourceProvider

• Dotel.java.disabled.resource.providers

  • io.opentelemetry.contrib.gcp.resource.GCPResourceProvider
  • io.opentelemetry.contrib.aws.resource.Ec2ResourceProvider

The way this worked:

• Setting otel.java.enabled.resource.providers causes only specified HostResourceProvider and ProcessResourceProvider to be active
• Setting otel.java.disabled.resource.providers overrides otel.java.enabled.resource.providers, but does nothing in this case because neither GCPResourceProvider or Ec2ResourceProvider are enabled

Hey @jack-berg thanks for the response. I originally went down this path because I noticed traces published at start up indicating the agent was attempting to call AWS/GCP metadata services. I picked those two providers from the list here: https://opentelemetry.io/docs/languages/java/configuration/#resourceprovider because I was trying to pick the minimal resource providers for a faster startup time, and figured I would add more if what they had wasn't sufficient.

Since enabled.resource.providers is a strict allow list, this behavior makes sense/isn't a bug - but I read the note and list here as being a comprehensive list of all possible resource providers rather than only those in -contrib, so I didn't realize I was disabling built in providers

[AlexeiZenin]

@laurit Ah - thank you, that appears to have been the fix, as after removing that line I do see service names in my traces! So basically I would need to add -Dotel.java.enabled.resource.providers=**io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider**,others ? Edit: Is it worth adding that to the docs here https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/languages/java/configuration.md ? Happy to raise a PR to do so

Just ran into the same thing today, never would have occurred to me a resource provider would prevent the OTEL service name attribute from being set and thus mangling under which service my traces/metrics would be routed to.

[trask]

I agree this is confusing. Luckily it looks like the (still experimental) declarative configuration is much clearer when it comes to including and excluding resource attributes: https://github.com/open-telemetry/opentelemetry-configuration/blob/60a2d91aa03b94b1bd34b85cc349a83ed9f35527/examples/kitchen-sink.yaml#L439-L455

you can start using declarative configuration already if you would like, here's an example with the java agent: https://github.com/open-telemetry/opentelemetry-java-examples/blob/main/javaagent/sdk-config.yaml

[AlexeiZenin]

Just tried this (as per the solution mentioned above) and it still didn't work, maybe there is another one I need for it to work with -D params?

-Dotel.java.enabled.resource.providers=io.opentelemetry.contrib.aws.resource.EcsResourceProvider,io.opentelemetry.instrumentation.resources.EnvironmentResourceProvider
-Dotel.service.name=alexei-test

I think I might just go for the exclude list now and remove the ones that add all the attributes I don't want (not ideal as I want to tightly control what attributes get added and don't want new ones appearing without us opting in)