Unable to launch OTel Java Agent from an uber JAR

[rvesse]

Describe the bug

In a similar vein to #8227 I've been experimenting with whether you can package the Otel Agent JAR into an uber JAR and have that work. I'm pretty sure the answer is no this won't work, so it would be nice if this were explicitly stated that this isn't supported somewhere in the documentation

Firstly if you're attempting to package into an uber jar you need to set both the Premain-Class and Launcher-Agent-Class attributes in manifest.mf. It's Launcher-Agent-Class that actually starts the agent in an uber JAR deployment BUT if you don't have Premain-Class as well OTel baulks with the following error:

ERROR io.opentelemetry.javaagent.OpenTelemetryAgent
java.lang.IllegalStateException: The agent was not installed, because the agent was found in 'example.jar', which doesn't contain a Premain-Class manifest attribute. Make sure that you haven't included the agent jar file inside of an application uber jar.

Clearly implying that the authors didn't expect this to be supported anyway.

If you do specify the Premain-Class attribute to force your way past this error you'll get the following error on startup:

OpenTelemetry Javaagent failed to start
java.lang.IllegalStateException: Could not install class file transformer
at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11253)
at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11155)
at net.bytebuddy.agent.builder.AgentBuilder$Default$Delegator.installOn(AgentBuilder.java:12927)
at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:181)
at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:94)
at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:78)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:35)
at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
at io.opentelemetry.javaagent.OpenTelemetryAgent.agentmain(OpenTelemetryAgent.java:49)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadAgent0(Native Method)
at java.instrument/sun.instrument.InstrumentationImpl.loadAgent(InstrumentationImpl.java:556)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/sun.launcher.LauncherHelper.lambda$getMainClassFromJar$0(LauncherHelper.java:576)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at java.base/sun.launcher.LauncherHelper.getMainClassFromJar(LauncherHelper.java:571)
at java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:778)
at java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:686)
Caused by: java.lang.UnsupportedOperationException: adding retransformable transformers is not supported in this environment
at java.instrument/sun.instrument.InstrumentationImpl.addTransformer(InstrumentationImpl.java:96)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at net.bytebuddy.utility.Invoker$Dispatcher.invoke(Unknown Source)
at net.bytebuddy.utility.dispatcher.JavaDispatcher$Dispatcher$ForNonStaticMethod.invoke(JavaDispatcher.java:1032)
at net.bytebuddy.utility.dispatcher.JavaDispatcher$ProxiedInvocationHandler.invoke(JavaDispatcher.java:1162)
at net.bytebuddy.agent.builder.$Proxy5.addTransformer(Unknown Source)
at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11230)
... 25 more

Which likely is the same underlying bug raphw/byte-buddy/issues/374 mentioned in the comments thread on #8227

Steps to reproduce

Generate an über JAR with the OTel Java Agent using the toolchain of your choice, e.g. Maven Shade plugin:

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <configuration>
          <shadedArtifactAttached>false</shadedArtifactAttached>
          <transformers>
            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
              <mainClass>my.package.MainClass</mainClass>
              <!-- https://issues.apache.org/jira/browse/LOG4J2-2537 -->
              <manifestEntries>
                <Multi-Release>true</Multi-Release>
                <Premain-Class>io.opentelemetry.javaagent.OpenTelemetryAgent</Premain-Class>
                <Launcher-Agent-Class>io.opentelemetry.javaagent.OpenTelemetryAgent</Launcher-Agent-Class>
              </manifestEntries>
            </transformer>
            <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
          </transformers>
          <filters>
            <filter>
              <artifact>*:*</artifact>
              <excludes>
                <!-- 
                     Some jars are signed but shading breaks that. 
                     Don't include signing files. 
                -->
                <exclude>META-INF/*.SF</exclude>
                <exclude>META-INF/*.DSA</exclude>
                <exclude>META-INF/*.RSA</exclude>
              </excludes>
            </filter>
          </filters>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

Expected behavior

N/A, seems this isn't intended to work/can't work so documentation merely needs to be clearer around this

Actual behavior

Errors thrown as shown earlier in description

Javaagent or library instrumentation version

1.24.0

Environment

JDK: Temurin 17 and OpenJDK 17
OS: Mac OX X Ventura, Linux (5.15.40-linuxkit)

Additional context

No response

[trask]

hi @rvesse! is this feature what you're looking for? https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/runtime-attach

[rvesse]

hi @rvesse! is this feature what you're looking for? https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/runtime-attach

No, I don't think that solves the problem for Uber JARs anyway, I get a slightly different error condition but my application still crashes:

[otel.javaagent 2023-10-17 15:34:29:703 +0100] [Attach Listener] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.20.1-SNAPSHOT
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

Personally I really don't want to put the agent into an uber JAR because that's not how agents are supposed to be used. But I'd ideally like some documentation saying Don't do that because X (or equivalent) in your official documentation so I point to that to justify why we aren't doing that.

[trask]

No, I don't think that solves the problem for Uber JARs anyway

can you describe a bit more why not? runtime attach module should allows you to embed the agent in an uber jar (e.g. spring boot)

[laurit]

You probably get the UnsupportedOperationException because you didn't add Can-Retransform-Classes: true manifest attribute, you should also add Can-Redefine-Classes: true. Packaging the agent into an uberjar like that will take some effort to make it work, I think it would be easier to get it running by packaging the agent as a jar file into your uberjar. In the Launcher-Agent-Class unpack that jar and use an URLClassLoader and a bit of reflection to call premain in OpenTelemetryAgent. I haven't tried it myself so can't say for sure whether this is all that is needed. If you want to package the agent into the uberjar then you have to be aware that in

opentelemetry-java-instrumentation/javaagent-bootstrap/src/main/java/io/opentelemetry/javaagent/OpenTelemetryAgent.java

Line 96 in 60b6548

inst.appendToBootstrapClassLoaderSearch(agentJar);

the agent adds the jar it is loaded from to boot class path, this may break your application.

[rvesse]

To summarise:

• Packaging the agent into an uber JAR is theoretically possible (if appropriate manifest.mf attributes are set) but likely a very bad idea per @laurit's comment as your entire application ends up on the bootstrap classpath which can lead to all sorts of subtle and interesting class loading problems
• Runtime Attach is an option (but at least for my use case/deployment environment doesn't work)

Note that I'm quite happy for you guys to close this as Won't Fix or your equivalent.

My main goal in filing this issue was to have some more easily discoverable discussion of this topic to help future engineers who get asked to try and do this and start Googling and don't waste too much time.

[alagusundarams]

@trask we are trying to use the agent with quarkus Uber jar and we end up getting the same exception as stated by @rvesse. Is this the limitation of otel agent with uber jars?

[trask]

hi @alagusundarams! if Runtime Attach isn't working for you, can you open an issue in https://github.com/open-telemetry/opentelemetry-java-contrib?

[breedx-splk]

Hi @alagusundarams and @rvesse -- did you open an issue in the contrib repo? If so, can you please mention it here? Thanks!

[github-actions]

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.