Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1 #24004

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Feb 19, 2025

Motivation

In Pulsar, we use ASF hosted Gradle Delocity for analysing Pulsar's maven builds.
Example listing of latest builds: https://develocity.apache.org/scans?search.rootProjectNames=Pulsar

There was a deadlock in https://github.com/apache/pulsar/actions/runs/13410107140/job/37458613913#step:13:1612 which could be caused by Gradle Develocity Maven Extension. The currently used version is 1.22.2 . Since there's a newer version available, it's worth upgrading to see if it resolves the issue in future builds.

Found one Java-level deadlock:
=============================
"main":
  waiting to lock monitor 0x00007f1a2c8ed3d0 (object 0x000010001b819ea8, a [Ljava.lang.Object;),
  which is held by "Thread-110"

"Thread-110":
  waiting to lock monitor 0x00007f1ae0002c60 (object 0x00001000073251b0, a org.apache.logging.log4j.core.appender.OutputStreamManager),
  which is held by "BookieDeathWatcher-40525"

"BookieDeathWatcher-40525":
  waiting to lock monitor 0x00007f1a58455960 (object 0x00001000085a0798, a com.gradle.maven.scan.extension.test.listener.obfuscated.g.b),
  which is held by "main"

Java stack information for the threads listed above:
===================================================
"main":
	at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:143)
	- waiting to lock <0x000010001b819ea8> (a [Ljava.lang.Object;)
	at org.testcontainers.DockerClientFactory.dockerHostIpAddress(DockerClientFactory.java:328)
	at org.testcontainers.containers.ContainerState.getHost(ContainerState.java:64)
	at io.etcd.jetcd.launcher.EtcdContainer.clientEndpoint(EtcdContainer.java:264)
	at io.etcd.jetcd.launcher.EtcdClusterImpl$$Lambda/0x00007f1a91101de8.apply(Unknown Source)
	at java.util.stream.ReferencePipeline$3$1.accept(java.base@21.0.6/ReferencePipeline.java:197)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@21.0.6/ArrayList.java:1708)
	at java.util.stream.AbstractPipeline.copyInto(java.base@21.0.6/AbstractPipeline.java:509)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@21.0.6/AbstractPipeline.java:499)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@21.0.6/ReduceOps.java:921)
	at java.util.stream.AbstractPipeline.evaluate(java.base@21.0.6/AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(java.base@21.0.6/ReferencePipeline.java:682)
	at io.etcd.jetcd.launcher.EtcdClusterImpl.clientEndpoints(EtcdClusterImpl.java:113)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest.getEtcdClusterConnectString(BaseMetadataStoreTest.java:185)
	- locked <0x000010001b8abe20> (a org.apache.pulsar.broker.EndToEndMetadataTest)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest.lambda$allImplementations$3(BaseMetadataStoreTest.java:142)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest$$Lambda/0x00007f1a9100a8f0.get(Unknown Source)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest$StringSupplier.get(BaseMetadataStoreTest.java:202)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest$StringSupplier.toString(BaseMetadataStoreTest.java:207)
	at java.lang.String.valueOf(java.base@21.0.6/String.java:4465)
	at com.gradle.maven.scan.extension.test.listener.testng.g.a(SourceFile:137)
	at com.gradle.maven.scan.extension.test.listener.testng.g.b(SourceFile:118)
	at com.gradle.maven.scan.extension.test.listener.testng.g.a(SourceFile:104)
	at com.gradle.maven.scan.extension.test.listener.testng.k.a(SourceFile:21)
	at com.gradle.maven.scan.extension.test.listener.testng.j.a(SourceFile:117)
	at com.gradle.maven.scan.extension.test.listener.testng.l.a(SourceFile:155)
	at com.gradle.maven.scan.extension.test.listener.testng.h.onTestStart(SourceFile:256)
	- locked <0x00001000085a0798> (a com.gradle.maven.scan.extension.test.listener.obfuscated.g.b)
	at com.gradle.maven.scan.extension.test.listener.testng.e.onTestStart(SourceFile:66)
	at com.gradle.maven.scan.extension.test.listener.testng.ExceptionWrappingTestNGListener.onTestStart(SourceFile:84)
	at org.testng.internal.TestListenerHelper.runTestListeners(TestListenerHelper.java:106)
	at org.testng.internal.invokers.TestInvoker.runTestResultListener(TestInvoker.java:277)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:638)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at org.testng.TestRunner$$Lambda/0x00007f1a9023f2f0.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@21.0.6/ArrayList.java:1596)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
"Thread-110":
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:365)
	- waiting to lock <0x00001000073251b0> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:101)
	at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:66)
	at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:67)
	at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:31)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:240)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2906)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2859)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2841)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2637)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2402)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:178)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.tryOutStrategy(DockerClientProviderStrategy.java:292)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$5(DockerClientProviderStrategy.java:265)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy$$Lambda/0x00007f1a910945e8.test(Unknown Source)
	at java.util.stream.ReferencePipeline$2$1.accept(java.base@21.0.6/ReferencePipeline.java:178)
	at java.util.stream.ReferencePipeline$2$1.accept(java.base@21.0.6/ReferencePipeline.java:179)
	at java.util.stream.ReferencePipeline$2$1.accept(java.base@21.0.6/ReferencePipeline.java:179)
	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(java.base@21.0.6/ArrayList.java:[1685](https://github.com/apache/pulsar/actions/runs/13410107140/job/37458613913#step:13:1686))
	at java.util.stream.ReferencePipeline.forEachWithCancel(java.base@21.0.6/ReferencePipeline.java:129)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(java.base@21.0.6/AbstractPipeline.java:527)
	at java.util.stream.AbstractPipeline.copyInto(java.base@21.0.6/AbstractPipeline.java:513)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@21.0.6/AbstractPipeline.java:499)
	at java.util.stream.FindOps$FindOp.evaluateSequential(java.base@21.0.6/FindOps.java:150)
	at java.util.stream.AbstractPipeline.evaluate(java.base@21.0.6/AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.findFirst(java.base@21.0.6/ReferencePipeline.java:647)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:266)
	at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:150)
	- locked <0x000010001b819ea8> (a [Ljava.lang.Object;)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:191)
	- locked <0x000010001b819ea8> (a [Ljava.lang.Object;)
	at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:104)
	at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:109)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:321)
	at io.etcd.jetcd.launcher.EtcdContainer.start(EtcdContainer.java:245)
	at io.etcd.jetcd.launcher.EtcdClusterImpl.lambda$start$2(EtcdClusterImpl.java:72)
	at io.etcd.jetcd.launcher.EtcdClusterImpl$$Lambda/0x00007f1a9103bb88.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
"BookieDeathWatcher-40525":
	at com.gradle.maven.scan.extension.test.listener.obfuscated.h.a.a(SourceFile:105)
	- waiting to lock <0x00001000085a0798> (a com.gradle.maven.scan.extension.test.listener.obfuscated.g.b)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.h.a.b(SourceFile:65)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.h.a$$Lambda/0x00007f1a90202808.text(Unknown Source)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.a.a(SourceFile:85)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.a.flush(SourceFile:78)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.a.write(SourceFile:72)
	at java.io.PrintStream.implWrite(java.base@21.0.6/PrintStream.java:643)
	at java.io.PrintStream.write(java.base@21.0.6/PrintStream.java:623)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.b.write(SourceFile:203)
	at org.apache.logging.log4j.core.appender.ConsoleAppender$SystemOutStream.write(ConsoleAppender.java:381)
	at java.io.PrintStream.implWrite(java.base@21.0.6/PrintStream.java:643)
	at java.io.PrintStream.write(java.base@21.0.6/PrintStream.java:623)
	at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:53)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:263)
	- eliminated <0x00001000073251b0> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:296)
	- eliminated <0x00001000073251b0> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:307)
	- locked <0x00001000073251b0> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:229)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2906)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2859)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2841)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2625)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2373)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:173)
	at org.apache.bookkeeper.proto.BookieServer$DeathWatcher.run(BookieServer.java:285)

Found 1 deadlock.

Modifications

Upgrade org.gradle:develocity-maven-extension in .mvn/extensions.xml to 1.23.1

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 4.1.0 milestone Feb 19, 2025
@lhotari lhotari self-assigned this Feb 19, 2025
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Feb 19, 2025
@lhotari
Copy link
Member Author

lhotari commented Feb 19, 2025

@clayburn Do you have a chance to check the deadlock details available in the issue description which contains references to Gradle Develocity Maven Extension?

@lhotari
Copy link
Member Author

lhotari commented Feb 19, 2025

@clayburn Another similar deadlock in https://github.com/apache/pulsar/actions/runs/13410108649/job/37458643828#step:13:794. Do you know if this is fixed with com.gradle:develocity-maven-extension:1.23.1 ?

Found one Java-level deadlock:
=============================
"main":
  waiting to lock monitor 0x00007f0955d2ebe0 (object 0x0000100005cb0d00, a org.apache.logging.log4j.core.appender.OutputStreamManager),
  which is held by "BookieDeathWatcher-33633"

"BookieDeathWatcher-33633":
  waiting to lock monitor 0x00007f06940429f0 (object 0x0000100032fb3198, a com.gradle.maven.scan.extension.test.listener.obfuscated.g.b),
  which is held by "main"

Java stack information for the threads listed above:
===================================================
"main":
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:365)
	- waiting to lock <0x0000100005cb0d00> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:101)
	at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:66)
	at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:67)
	at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:31)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:240)
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2906)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2859)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2841)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2637)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2402)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:178)
	at org.testcontainers.utility.ImageNameSubstitutor.instance(ImageNameSubstitutor.java:60)
	- locked <0x000010003f263640> (a java.lang.Class for org.testcontainers.utility.ImageNameSubstitutor)
	at org.testcontainers.images.RemoteDockerImage.<init>(RemoteDockerImage.java:41)
	at org.testcontainers.images.RemoteDockerImage.<init>(RemoteDockerImage.java:52)
	at org.testcontainers.containers.GenericContainer.setDockerImageName(GenericContainer.java:1396)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:258)
	at io.etcd.jetcd.launcher.EtcdContainer.<init>(EtcdContainer.java:68)
	at io.etcd.jetcd.launcher.EtcdClusterImpl.lambda$new$1(EtcdClusterImpl.java:59)
	at io.etcd.jetcd.launcher.EtcdClusterImpl$$Lambda/0x00007f06d9002240.apply(Unknown Source)
	at java.util.stream.ReferencePipeline$3$1.accept(java.base@21.0.6/ReferencePipeline.java:197)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@21.0.6/ArrayList.java:1708)
	at java.util.stream.AbstractPipeline.copyInto(java.base@21.0.6/AbstractPipeline.java:509)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@21.0.6/AbstractPipeline.java:499)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@21.0.6/ReduceOps.java:921)
	at java.util.stream.AbstractPipeline.evaluate(java.base@21.0.6/AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(java.base@21.0.6/ReferencePipeline.java:682)
	at io.etcd.jetcd.launcher.EtcdClusterImpl.<init>(EtcdClusterImpl.java:61)
	at io.etcd.jetcd.launcher.Etcd$Builder.build(Etcd.java:116)
	at io.etcd.jetcd.test.EtcdClusterExtension$Builder.build(EtcdClusterExtension.java:172)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest.getEtcdClusterConnectString(BaseMetadataStoreTest.java:181)
	- locked <0x0000100045d9aa60> (a org.apache.pulsar.broker.EndToEndMetadataTest)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest.lambda$allImplementations$3(BaseMetadataStoreTest.java:142)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest$$Lambda/0x00007f06d8bac000.get(Unknown Source)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest$StringSupplier.get(BaseMetadataStoreTest.java:202)
	at org.apache.pulsar.metadata.BaseMetadataStoreTest$StringSupplier.toString(BaseMetadataStoreTest.java:207)
	at java.lang.String.valueOf(java.base@21.0.6/String.java:4465)
	at com.gradle.maven.scan.extension.test.listener.testng.g.a(SourceFile:137)
	at com.gradle.maven.scan.extension.test.listener.testng.g.b(SourceFile:118)
	at com.gradle.maven.scan.extension.test.listener.testng.g.a(SourceFile:104)
	at com.gradle.maven.scan.extension.test.listener.testng.k.a(SourceFile:21)
	at com.gradle.maven.scan.extension.test.listener.testng.j.a(SourceFile:117)
	at com.gradle.maven.scan.extension.test.listener.testng.l.a(SourceFile:155)
	at com.gradle.maven.scan.extension.test.listener.testng.h.onTestStart(SourceFile:256)
	- locked <0x0000100032fb3198> (a com.gradle.maven.scan.extension.test.listener.obfuscated.g.b)
	at com.gradle.maven.scan.extension.test.listener.testng.e.onTestStart(SourceFile:66)
	at com.gradle.maven.scan.extension.test.listener.testng.ExceptionWrappingTestNGListener.onTestStart(SourceFile:84)
	at org.testng.internal.TestListenerHelper.runTestListeners(TestListenerHelper.java:106)
	at org.testng.internal.invokers.TestInvoker.runTestResultListener(TestInvoker.java:277)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:638)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at org.testng.TestRunner$$Lambda/0x00007f06d82c3cb0.accept(Unknown Source)
	at java.util.ArrayList.forEach(java.base@21.0.6/ArrayList.java:1596)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
"BookieDeathWatcher-33633":
	at com.gradle.maven.scan.extension.test.listener.obfuscated.h.a.a(SourceFile:105)
	- waiting to lock <0x0000100032fb3198> (a com.gradle.maven.scan.extension.test.listener.obfuscated.g.b)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.h.a.b(SourceFile:65)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.h.a$$Lambda/0x00007f06d[820](https://github.com/apache/pulsar/actions/runs/13410108649/job/37458643828#step:13:821)9308.text(Unknown Source)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.a.a(SourceFile:85)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.a.flush(SourceFile:78)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.a.write(SourceFile:72)
	at java.io.PrintStream.implWrite(java.base@21.0.6/PrintStream.java:643)
	at java.io.PrintStream.write(java.base@21.0.6/PrintStream.java:623)
	at com.gradle.maven.scan.extension.test.listener.obfuscated.l.b.write(SourceFile:203)
	at org.apache.logging.log4j.core.appender.ConsoleAppender$SystemOutStream.write(ConsoleAppender.java:381)
	at java.io.PrintStream.implWrite(java.base@21.0.6/PrintStream.java:643)
	at java.io.PrintStream.write(java.base@21.0.6/PrintStream.java:623)
	at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:53)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:263)
	- eliminated <0x0000100005cb0d00> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:296)
	- locked <0x0000100005cb0d00> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:307)
	- locked <0x0000100005cb0d00> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:229)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
	at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2906)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2859)
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2[841](https://github.com/apache/pulsar/actions/runs/13410108649/job/37458643828#step:13:842))
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2625)
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2373)
	at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:173)
	at org.apache.bookkeeper.proto.BookieServer$DeathWatcher.run(BookieServer.java:285)

Found 1 deadlock.

@lhotari
Copy link
Member Author

lhotari commented Feb 19, 2025

Found Develocity Maven Extension release notes at https://docs.gradle.com/develocity/maven-extension/current/#release_history . There doesn't seem to be recent deadlock fixes.

@clayburn
Copy link
Contributor

Thanks for raising this. I'm not aware of any fix in 1.23.1, so we will investigate and come back to you.

@lhotari
Copy link
Member Author

lhotari commented Feb 19, 2025

The problem seems to consistently reproduce. For example it failed in the build of this PR: https://github.com/apache/pulsar/actions/runs/13413883514/job/37470450862?pr=24004#step:13:857

@lhotari
Copy link
Member Author

lhotari commented Feb 19, 2025

Since Testcontainers seems to always be part of the stack traces, I have a PR to upgrade to latest Testcontainers version in #24003, just to rule out any known issues related to Testcontainers that are fixed in latest.

@pshevche
Copy link

@lhotari , I looked into this issue on the Develocity Maven extension side. I think I have something that we can try to prevent the deadlock from happening.

When your build is running, we capture info about the TestNG tests as they start. If the test uses a DataProvider, we will evaluate the toString value of the parameter to show meaningful iteration names in Build Scan. It looks like one of the toString methods causes this deadlock. In particular, all of the stacktraces you shared have this common part (the following snippet is from this comment):

"main":
        at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:143)
        - waiting to lock <0x0000100034234678> (a [Ljava.lang.Object;)
        at org.testcontainers.DockerClientFactory.dockerHostIpAddress(DockerClientFactory.java:328)
        at org.testcontainers.containers.ContainerState.getHost(ContainerState.java:64)
        at io.etcd.jetcd.launcher.EtcdContainer.clientEndpoint(EtcdContainer.java:264)
        at io.etcd.jetcd.launcher.EtcdClusterImpl$$Lambda/0x00007f467101ed80.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at io.etcd.jetcd.launcher.EtcdClusterImpl.clientEndpoints(EtcdClusterImpl.java:113)
        // toString supplier for the parameter
        at org.apache.pulsar.metadata.BaseMetadataStoreTest.getEtcdClusterConnectString(BaseMetadataStoreTest.java:185) 
        - locked <0x0000100035f33218> (a org.apache.pulsar.broker.EndToEndMetadataTest)
        // data provider definition
        at org.apache.pulsar.metadata.BaseMetadataStoreTest.lambda$allImplementations$3(BaseMetadataStoreTest.java:142) 
        at org.apache.pulsar.metadata.BaseMetadataStoreTest$$Lambda/0x00007f4670bfe800.get(Unknown Source)
        at org.apache.pulsar.metadata.BaseMetadataStoreTest$StringSupplier.get(BaseMetadataStoreTest.java:202)
        at org.apache.pulsar.metadata.BaseMetadataStoreTest$StringSupplier.toString(BaseMetadataStoreTest.java:207)
        at java.lang.String.valueOf(String.java:4465)
        // here Develocity extension tries to get a string representation of the parameter.

At the same time, we see that some other thread is already executing DockerClientFactory#getorInitializeStrategy, which seems to cause this deadlock:

"Thread-140":
        ...
        at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:150)
        - locked <0x0000100034234678> (a [Ljava.lang.Object;)
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:191)
        - locked <0x0000100034234678> (a [Ljava.lang.Object;)
        at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:104)
        at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:109)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:321)
        at io.etcd.jetcd.launcher.EtcdContainer.start(EtcdContainer.java:245)
        at io.etcd.jetcd.launcher.EtcdClusterImpl.lambda$start$2(EtcdClusterImpl.java:72)
        at io.etcd.jetcd.launcher.EtcdClusterImpl$$Lambda/0x00007f4670f58b18.run(Unknown Source)
        at java.lang.Thread.runWith(Thread.java:1596)
        at java.lang.Thread.run(Thread.java:1583)

Is there any way to simplify this parameter name supplier to not invoke the Testcontainers API (the clientEndpoints call here will do that)?

I am not 100% sure that it will help. For the sake of experiments, you could set it to some dumb value and see if the deadlock still happens at the same place. The deadlock stacktrace will point to the parameter value that causes it. It can happen that there are other parameter values that will cause the deadlock. Let us know if this helps.

@lhotari
Copy link
Member Author

lhotari commented Feb 21, 2025

@lhotari , I looked into this issue on the Develocity Maven extension side. I think I have something that we can try to prevent the deadlock from happening.

When your build is running, we capture info about the TestNG tests as they start. If the test uses a DataProvider, we will evaluate the toString value of the parameter to show meaningful iteration names in Build Scan. It looks like one of the toString methods causes this deadlock. In particular, all of the stacktraces you shared have this common part (the following snippet is from this comment):

Thank you for pointing out the problem, @pshevche. I'll make changes in the test class to address the issue so that the particular toString() implemention wouldn't eagerly trigger Testcontainers initialization which causes the deadlock in this case.

@lhotari
Copy link
Member Author

lhotari commented Feb 21, 2025

Workaround applied in 51563a1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants