Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Registry Deadlock during Location registration. #146

Open
DivineThreepwood opened this issue Oct 1, 2024 · 1 comment
Open

Registry Deadlock during Location registration. #146

DivineThreepwood opened this issue Oct 1, 2024 · 1 comment
Labels
bugfix

Comments

@DivineThreepwood
Copy link
Member

The following deadlock was reported:

Found one Java-level deadlock:
=============================
"pool-1-thread-3":
  waiting for ownable synchronizer 0x000000060175c6c0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "pool-1-thread-4"

"pool-1-thread-4":
  waiting to lock monitor 0x000060000266b8e0 (object 0x0000000602d7d1a0, a org.openbase.jul.schedule.SyncObject),
  which is held by "pool-1-thread-55"

"pool-1-thread-55":
  waiting for ownable synchronizer 0x0000000602ae1dd8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "pool-1-thread-4"

Java stack information for the threads listed above:
===================================================
"pool-1-thread-3":
	at jdk.internal.misc.Unsafe.park(java.base@21.0.4/Native Method)
	- parking to wait for  <0x000000060175c6c0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.4/LockSupport.java:221)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.4/AbstractQueuedSynchronizer.java:754)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@21.0.4/AbstractQueuedSynchronizer.java:1099)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lockInterruptibly(java.base@21.0.4/ReentrantReadWriteLock.java:783)
	at org.openbase.jul.storage.registry.AbstractRegistry.get(AbstractRegistry.java:457)
	at org.openbase.jul.storage.registry.ProtoBufFileSynchronizedRegistry.getMessage(ProtoBufFileSynchronizedRegistry.java:189)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.getUnitConfigById(UnitRegistryController.java:568)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.lambda$filterDataForUser$14(UnitRegistryController.java:1006)
	at org.openbase.bco.registry.unit.core.UnitRegistryController$$Lambda/0x0000000800667ca8.match(Unknown Source)
	at org.openbase.jul.pattern.ListFilter.filter(ListFilter.java:40)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.filterDataForUser(UnitRegistryController.java:1037)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.filterDataForUser(UnitRegistryController.java:97)
	at org.openbase.bco.authentication.lib.com.AbstractAuthenticatedControllerServer.updateDataToPublish(AbstractAuthenticatedControllerServer.java:144)
	at org.openbase.bco.authentication.lib.com.AbstractAuthenticatedControllerServer.requestStatus(AbstractAuthenticatedControllerServer.java:86)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeVirtual(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x0000000800134c00.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.4/Invokers$Holder)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base@21.0.4/DirectMethodHandleAccessor.java:153)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base@21.0.4/DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke(java.base@21.0.4/Method.java:580)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method.callMethod(CallerImpl.kt:97)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method$Instance.call(CallerImpl.kt:113)
	at kotlin.reflect.jvm.internal.KCallableImpl.call(KCallableImpl.kt:108)
	at org.openbase.jul.communication.mqtt.RPCMethodWrapper.invoke(RPCMethodWrapper.kt:224)
	at org.openbase.jul.communication.mqtt.RPCServerImpl.handleRemoteCall(RPCServerImpl.kt:159)
	at org.openbase.jul.communication.mqtt.RPCServerImpl.access$handleRemoteCall(RPCServerImpl.kt:33)
	at org.openbase.jul.communication.mqtt.RPCServerImpl$activate$1$2$1$1.invokeSuspend(RPCServerImpl.kt:84)
	at org.openbase.jul.communication.mqtt.RPCServerImpl$activate$1$2$1$1.invoke(RPCServerImpl.kt)
	at org.openbase.jul.communication.mqtt.RPCServerImpl$activate$1$2$1$1.invoke(RPCServerImpl.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:72)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:148)
	at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:43)
	at org.openbase.jul.communication.mqtt.RPCServerImpl$activate$1$2$1.invokeSuspend(RPCServerImpl.kt:83)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.4/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.4/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)
"pool-1-thread-4":
	at org.openbase.bco.registry.unit.core.plugin.AliasMapUpdatePlugin.afterRegister(AliasMapUpdatePlugin.java:56)
	- waiting to lock <0x0000000602d7d1a0> (a org.openbase.jul.schedule.SyncObject)
	at org.openbase.bco.registry.unit.core.plugin.AliasMapUpdatePlugin.afterRegister(AliasMapUpdatePlugin.java:37)
	at org.openbase.jul.storage.registry.plugin.RegistryPluginPool.afterRegister(RegistryPluginPool.java:144)
	at org.openbase.jul.storage.registry.AbstractRegistry.register(AbstractRegistry.java:198)
	at org.openbase.jul.storage.registry.FileSynchronizedRegistryImpl.register(FileSynchronizedRegistryImpl.java:158)
	at org.openbase.jul.storage.registry.ProtoBufFileSynchronizedRegistry.register(ProtoBufFileSynchronizedRegistry.java:163)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.lambda$registerUnitConfigAuthenticated$4(UnitRegistryController.java:555)
	at org.openbase.bco.registry.unit.core.UnitRegistryController$$Lambda/0x0000000800c0ae88.process(Unknown Source)
	at org.openbase.bco.authentication.lib.AuthenticatedServiceProcessor.authenticatedAction(AuthenticatedServiceProcessor.java:134)
	at org.openbase.bco.authentication.lib.AuthenticatedServiceProcessor.authenticatedAction(AuthenticatedServiceProcessor.java:90)
	at org.openbase.bco.authentication.lib.AuthenticatedServiceProcessor.authenticatedAction(AuthenticatedServiceProcessor.java:69)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.lambda$registerUnitConfigAuthenticated$5(UnitRegistryController.java:541)
	at org.openbase.bco.registry.unit.core.UnitRegistryController$$Lambda/0x0000000800c0ac68.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(java.base@21.0.4/FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.4/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.4/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)
"pool-1-thread-55":
	at jdk.internal.misc.Unsafe.park(java.base@21.0.4/Native Method)
	- parking to wait for  <0x0000000602ae1dd8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.4/LockSupport.java:221)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.4/AbstractQueuedSynchronizer.java:754)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@21.0.4/AbstractQueuedSynchronizer.java:1099)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lockInterruptibly(java.base@21.0.4/ReentrantReadWriteLock.java:783)
	at org.openbase.jul.storage.registry.AbstractRegistry.get(AbstractRegistry.java:457)
	at org.openbase.jul.storage.registry.ProtoBufFileSynchronizedRegistry.getMessage(ProtoBufFileSynchronizedRegistry.java:189)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.getUnitConfigById(UnitRegistryController.java:568)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.getUnitConfigByAlias(UnitRegistryController.java:590)
	- locked <0x0000000602d7d1a0> (a org.openbase.jul.schedule.SyncObject)
	at org.openbase.bco.registry.unit.lib.auth.AuthorizationWithTokenHelper.canDo(AuthorizationWithTokenHelper.kt:237)
	at org.openbase.bco.registry.unit.lib.auth.AuthorizationWithTokenHelper.canDo$default(AuthorizationWithTokenHelper.kt:206)
	at org.openbase.bco.registry.unit.lib.auth.AuthorizationWithTokenHelper.canDo(AuthorizationWithTokenHelper.kt)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.lambda$registerUnitConfigAuthenticated$4(UnitRegistryController.java:552)
	at org.openbase.bco.registry.unit.core.UnitRegistryController$$Lambda/0x0000000800c0ae88.process(Unknown Source)
	at org.openbase.bco.authentication.lib.AuthenticatedServiceProcessor.authenticatedAction(AuthenticatedServiceProcessor.java:134)
	at org.openbase.bco.authentication.lib.AuthenticatedServiceProcessor.authenticatedAction(AuthenticatedServiceProcessor.java:90)
	at org.openbase.bco.authentication.lib.AuthenticatedServiceProcessor.authenticatedAction(AuthenticatedServiceProcessor.java:69)
	at org.openbase.bco.registry.unit.core.UnitRegistryController.lambda$registerUnitConfigAuthenticated$5(UnitRegistryController.java:541)
	at org.openbase.bco.registry.unit.core.UnitRegistryController$$Lambda/0x0000000800c0ac68.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(java.base@21.0.4/FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.4/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.4/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)

Found 1 deadlock.

Heap
 garbage-first heap   total 184320K, used 113750K [0x0000000600000000, 0x0000000800000000)
  region size 4096K, 12 young (49152K), 3 survivors (12288K)
 Metaspace       used 103223K, committed 104064K, reserved 1179648K
  class space    used 12017K, committed 12416K, reserved 1048576K
@DivineThreepwood
Copy link
Member Author

Its was cause by a fast registration of locations. Can pretty sure be reproduced by registering multiple locationConfigs at once in a loop without waiting for the future to be returned. I guess the registry is just not thread safe :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bugfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DivineThreepwood