Possible deadlock with ConcurrentStoreAndDispatch with useCache=true

[mattrpav]

Version: v5.19.2 (w/ recent KahaDB updates)

Notes: this only occurs when useCache=true. When disabled, there is no deadlock.

Producer thread:

   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.2/Native Method)
	- parking to wait for  <0x000000078b900220> (a org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask$InnerFutureTask)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.2/LockSupport.java:221)
	at java.util.concurrent.FutureTask.awaitDone(java.base@21.0.2/FutureTask.java:500)
	at java.util.concurrent.FutureTask.get(java.base@21.0.2/FutureTask.java:190)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.waitForAsyncMessage(AbstractStoreCursor.java:575)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.syncWithStore(AbstractStoreCursor.java:337)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.tryAddMessageLast(AbstractStoreCursor.java:269)
	- locked <0x000000078b900260> (a org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
	at org.apache.activemq.broker.region.cursors.AbstractPendingMessageCursor.addMessageLast(AbstractPendingMessageCursor.java:93)
	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.tryAddMessageLast(StoreQueueCursor.java:105)
	- locked <0x000000078b900398> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
	at org.apache.activemq.broker.region.cursors.AbstractPendingMessageCursor.addMessageLast(AbstractPendingMessageCursor.java:93)
	at org.apache.activemq.broker.region.Queue.doPendingCursorAdditions(Queue.java:827)
	at org.apache.activemq.broker.region.Queue.tryOrderedCursorAdd(Queue.java:938)
	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:917)
	at org.apache.activemq.broker.region.Queue.send(Queue.java:767)
...
   Locked ownable synchronizers:
        - ...
        - ...
	- <0x00000007bf78a970> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

ConcurrentQueueStoreAndDispatch thread:

"ConcurrentQueueStoreAndDispatch" #657 [122627] daemon prio=8 os_prio=31 cpu=890.98ms elapsed=815.63s tid=0x000000014d982600 nid=122627 waiting on condition  [0x000000035faca000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.2/Native Method)
	- parking to wait for  <0x00000007bf78a970> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.2/LockSupport.java:221)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.2/AbstractQueuedSynchronizer.java:754)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21.0.2/AbstractQueuedSynchronizer.java:990)
	at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21.0.2/ReentrantLock.java:153)
	at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.2/ReentrantLock.java:322)
	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1264)
	at org.apache.activemq.store.kahadb.MessageDatabase$6.visit(MessageDatabase.java:1198)
	at org.apache.activemq.store.kahadb.data.KahaAddMessageCommand.visit(KahaAddMessageCommand.java:241)
	at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:1195)
	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1089)
	at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1065)
	at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:537)
	at org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask.run(KahaDBStore.java:1669)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.2/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.2/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.2/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.2/Thread.java:1583)

   Locked ownable synchronizers:
	- <0x00000007bf614ef0> (a java.util.concurrent.ThreadPoolExecutor$Worker)