Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Offload xmpp/smack processing threads. #500

Merged
merged 4 commits into from
Sep 27, 2023
Merged

Conversation

damencho
Copy link
Member

Make sure we do not send and/or wait for responses in the smack threads.

Make sure we do not send and/or wait for responses in the smack threads.
@codecov
Copy link

codecov bot commented Sep 25, 2023

Codecov Report

Merging #500 (778ba0b) into master (7ae8def) will increase coverage by 0.20%.
The diff coverage is 30.61%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #500      +/-   ##
============================================
+ Coverage     20.82%   21.02%   +0.20%     
- Complexity      286      292       +6     
============================================
  Files            74       74              
  Lines          6186     6220      +34     
  Branches        832      832              
============================================
+ Hits           1288     1308      +20     
- Misses         4675     4690      +15     
+ Partials        223      222       -1     
Files Coverage Δ
.../jitsi/jigasi/sounds/SoundNotificationManager.java 30.04% <50.00%> (ø)
...org/jitsi/jigasi/xmpp/CallControlMucActivator.java 14.07% <0.00%> (-0.11%) ⬇️
src/main/java/org/jitsi/jigasi/lobby/Lobby.java 0.00% <0.00%> (ø)
src/main/java/org/jitsi/jigasi/JvbConference.java 43.60% <34.14%> (+0.82%) ⬆️

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ae8def...778ba0b. Read the comment docs.

@damencho
Copy link
Member Author

"PipelineThread-0" #218 daemon prio=5 os_prio=0 cpu=6868.78ms elapsed=322209.08s tid=0x00007f4d68011800 nid=0x1e7a waiting for monitor entry  [0x00007f4d1f5fb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jivesoftware.smackx.muc.MultiUserChat.userHasLeft(MultiUserChat.java:2207)
	- waiting to lock <0x0000000741200000> (a org.jivesoftware.smackx.muc.MultiUserChat)
	at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.20/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.20/Method.java:566)
	at net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl.leave(ChatRoomJabberImpl.java:1078)
	at org.jitsi.jigasi.JvbConference.leaveConferenceRoom(JvbConference.java:1047)
	at org.jitsi.jigasi.JvbConference.stop(JvbConference.java:536)
	- locked <0x0000000741200308> (a org.jitsi.jigasi.JvbConference)
	at org.jitsi.jigasi.SipGatewaySession.sipCallEnded(SipGatewaySession.java:719)
	at org.jitsi.jigasi.SipGatewaySession$SipCallStateListener.handleCallState(SipGatewaySession.java:1499)
	at org.jitsi.jigasi.SipGatewaySession$SipCallStateListener.callStateChanged(SipGatewaySession.java:1441)
	at net.java.sip.communicator.service.protocol.Call.fireCallChangeEvent(Call.java:362)
	at net.java.sip.communicator.service.protocol.Call.setCallState(Call.java:408)
	at net.java.sip.communicator.service.protocol.media.MediaAwareCall.setCallState(MediaAwareCall.java:834)
	at net.java.sip.communicator.impl.protocol.sip.CallSipImpl.setCallState(CallSipImpl.java:770)
	at net.java.sip.communicator.service.protocol.media.MediaAwareCall.removeCallPeer(MediaAwareCall.java:222)
	at net.java.sip.communicator.service.protocol.media.MediaAwareCall.peerStateChanged(MediaAwareCall.java:283)
	at net.java.sip.communicator.service.protocol.AbstractCallPeer.fireCallPeerChangeEvent(AbstractCallPeer.java:426)
	at net.java.sip.communicator.service.protocol.AbstractCallPeer.setState(AbstractCallPeer.java:957)
	at net.java.sip.communicator.service.protocol.media.MediaAwareCallPeer.setState(MediaAwareCallPeer.java:1001)
	- locked <0x00000007414ca038> (a net.java.sip.communicator.impl.protocol.sip.CallPeerSipImpl$1)
	at net.java.sip.communicator.service.protocol.AbstractCallPeer.setState(AbstractCallPeer.java:929)
	at net.java.sip.communicator.service.protocol.AbstractCallPeer.setState(AbstractCallPeer.java:915)
	at net.java.sip.communicator.impl.protocol.sip.CallPeerSipImpl.processBye(CallPeerSipImpl.java:612)
	at net.java.sip.communicator.impl.protocol.sip.OperationSetBasicTelephonySipImpl.processBye(OperationSetBasicTelephonySipImpl.java:1211)
	at net.java.sip.communicator.impl.protocol.sip.OperationSetBasicTelephonySipImpl.processRequest(OperationSetBasicTelephonySipImpl.java:363)
	at net.java.sip.communicator.impl.protocol.sip.ProtocolProviderServiceSipImpl.processRequest(ProtocolProviderServiceSipImpl.java:1120)
	at net.java.sip.communicator.impl.protocol.sip.SipStackSharing.processRequest(SipStackSharing.java:699)
	at gov.nist.javax.sip.EventScanner.deliverRequestEvent(EventScanner.java:250)
	at gov.nist.javax.sip.EventScanner.deliverEvent(EventScanner.java:146)
	at gov.nist.javax.sip.SipProviderImpl.handleEvent(SipProviderImpl.java:185)
	at gov.nist.javax.sip.DialogFilter.processRequest(DialogFilter.java:1324)
	at gov.nist.javax.sip.stack.SIPServerTransactionImpl.processRequest(SIPServerTransactionImpl.java:811)
	at gov.nist.javax.sip.stack.ConnectionOrientedMessageChannel.processMessage(ConnectionOrientedMessageChannel.java:466)
	at gov.nist.javax.sip.parser.PipelinedMsgParser.run(PipelinedMsgParser.java:541)
	at java.lang.Thread.run(java.base@11.0.20/Thread.java:829)

"Smack Cached Executor" #112837 daemon prio=5 os_prio=0 cpu=2525.16ms elapsed=263450.56s tid=0x00007f4d38067800 nid=0x454af waiting for monitor entry  [0x00007f4cc72bc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jivesoftware.smackx.muc.MultiUserChat.userHasLeft(MultiUserChat.java:2207)
	- waiting to lock <0x0000000741200000> (a org.jivesoftware.smackx.muc.MultiUserChat)
	at org.jivesoftware.smackx.muc.MultiUserChat.access$900(MultiUserChat.java:116)
	at org.jivesoftware.smackx.muc.MultiUserChat$3.processStanza(MultiUserChat.java:262)
	at org.jivesoftware.smack.AbstractXMPPConnection.lambda$invokeStanzaCollectorsAndNotifyRecvListeners$8(AbstractXMPPConnection.java:1619)
	at org.jivesoftware.smack.AbstractXMPPConnection$$Lambda$321/0x00000008404d0840.run(Unknown Source)
	at org.jivesoftware.smack.AsyncButOrdered$Handler.run(AsyncButOrdered.java:151)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.20/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.20/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.20/Thread.java:829)

"Smack Cached Executor" #112858 daemon prio=5 os_prio=0 cpu=13.24ms elapsed=263441.89s tid=0x00007f4d34083800 nid=0x454c5 in Object.wait()  [0x00007f4ca2bbb000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.20/Native Method)
	- waiting on <no object reference available>
	at java.lang.Object.wait(java.base@11.0.20/Object.java:328)
	at org.jivesoftware.smackx.muc.MultiUserChat.enter(MultiUserChat.java:427)
	- waiting to re-lock in wait() <0x00000007412b9cb0> (a org.jivesoftware.smackx.muc.MultiUserChat$3)
	at org.jivesoftware.smackx.muc.MultiUserChat.join(MultiUserChat.java:721)
	- locked <0x0000000741200000> (a org.jivesoftware.smackx.muc.MultiUserChat)
	at org.jivesoftware.smackx.muc.MultiUserChat.join(MultiUserChat.java:651)
	at net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl.joinAs(ChatRoomJabberImpl.java:740)
	at net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl.joinAs(ChatRoomJabberImpl.java:874)
	at org.jitsi.jigasi.JvbConference.joinConferenceRoom(JvbConference.java:825)
	at org.jitsi.jigasi.lobby.Lobby.invitationReceived(Lobby.java:217)
	at net.java.sip.communicator.service.protocol.AbstractOperationSetMultiUserChat.fireInvitationReceived(AbstractOperationSetMultiUserChat.java:125)
	at net.java.sip.communicator.impl.protocol.jabber.OperationSetMultiUserChatJabberImpl.fireInvitationEvent(OperationSetMultiUserChatJabberImpl.java:732)
	at net.java.sip.communicator.impl.protocol.jabber.OperationSetMultiUserChatJabberImpl$SmackInvitationListener.invitationReceived(OperationSetMultiUserChatJabberImpl.java:770)
	at org.jivesoftware.smackx.muc.MultiUserChatManager$2.processStanza(MultiUserChatManager.java:195)
	at org.jivesoftware.smack.AbstractXMPPConnection$4.run(AbstractXMPPConnection.java:1601)
	at org.jivesoftware.smack.AbstractXMPPConnection$10.run(AbstractXMPPConnection.java:2146)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.20/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.20/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.20/Thread.java:829)

More discussions: https://discourse.igniterealtime.org/t/thread-stuck-in-multiuserchat-enter-forever/92090/15?u=flow
Jicofo's fix: https://github.com/jitsi/jicofo/pull/1082/files

@@ -603,7 +612,7 @@ private synchronized void setXmppProvider(

if (xmppProvider.isRegistered())
{
joinConferenceRoom();
xmppExecutorPool.execute(this::joinConferenceRoom);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might swallow exceptions

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

joinConferenceRoom() itself has one big try catch, that should catch everything...

* A thread pool used to offload xmpp execution in a new thread to avoid blocking
* xmpp threads.
*/
public static final ExecutorService xmppExecutorPool = Util.createNewThreadPool("xmpp-executor-pool");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not have a generic "io" thread pool in jigasi?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing, this becomes one for the xmpp handling.

@@ -1081,6 +1100,11 @@ public void serviceChanged(ServiceEvent serviceEvent)

@Override
public void memberPresenceChanged(ChatRoomMemberPresenceChangeEvent evt)
{
xmppExecutorPool.execute(() -> memberPresenceChangedInternal(evt));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These may now execute concurrently or out of order. Do they need to go on a queue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that matters ...

@@ -624,7 +624,7 @@ public void notifyChatRoomMemberJoined(ChatRoomMember member)
public void notifyChatRoomMemberLeft(ChatRoomMember member)
{
// if this is the sip hanging up (stopping) skip playing
if (gatewaySession.getJvbConference().isStarted()
if (gatewaySession.getJvbConference() != null && gatewaySession.getJvbConference().isStarted()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AbstractGatewaySession.jvbConference can change to null, so this can still produce an NPE (granted, rarely)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@damencho damencho merged commit b9e9572 into master Sep 27, 2023
4 checks passed
@damencho damencho deleted the fix-smack-bug branch September 27, 2023 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants