failed: ForkJoinPoolStarvationSpec on JDK 17

[patriknw]

https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3680

https://github.com/akka/akka/runs/5118048171?check_suite_focus=true#step:5:3526

https://github.com/akka/akka/runs/5086535435?check_suite_focus=true#step:5:3591

https://github.com/akka/akka/runs/5086535594?check_suite_focus=true#step:5:3594

and probably many more

[info] - must not starve tasks arriving from external dispatchers under high internal traffic *** FAILED *** (6 seconds, 13 milliseconds)
[3681](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3681)
[info]   java.lang.AssertionError: assertion failed: timeout (6 seconds) during expectMsg while waiting for All fine
[3682](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3682)
[info]   at scala.Predef$.assert(Predef.scala:223)
[3683](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3683)
[info]   at akka.testkit.TestKitBase.expectMsg_internal(TestKit.scala:461)
[3684](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3684)
[info]   at akka.testkit.TestKitBase.expectMsg(TestKit.scala:438)
[3685](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3685)
[info]   at akka.testkit.TestKitBase.expectMsg$(TestKit.scala:438)
[3686](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3686)
[info]   at akka.testkit.TestKit.expectMsg(TestKit.scala:973)
[3687](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3687)
[info]   at akka.dispatch.ForkJoinPoolStarvationSpec.$anonfun$new$6(ForkJoinPoolStarvationSpec.scala:64)
[3688](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3688)
[info]   at akka.dispatch.ForkJoinPoolStarvationSpec.$anonfun$new$6$adapted(ForkJoinPoolStarvationSpec.scala:61)
[3689](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3689)
[info]   at scala.collection.immutable.Range.foreach(Range.scala:158)
[3690](https://github.com/akka/akka/runs/5118047872?check_suite_focus=true#step:5:3690)
[info]   at akka.dispatch.ForkJoinPoolStarvationSpec.$anonfun$new$2(ForkJoinPoolStarvationSpec.scala:61)

[jrudolph]

https://github.com/akka/akka/runs/5133864147?check_suite_focus=true

[jrudolph]

It can be reproduced locally and, indeed, it seems external submissions seem to be starved in this version of the ForkJoinPool.

[jrudolph]

I tried finding out what changed with ForkJoinPool in JDK 17. The answer is "Yes" and it could mean anything. E.g. openjdk/jdk17u-dev@5cfa8c9?w=1#diff-e398beb49cd8d3e6c2f3a8ca8eee97172c57d7f88f3ccd8a3c704632cab32f5f

[jrudolph]

See #17413 for more info why this was added in the first place.

[jrudolph]

It seems ForkJoinPool does not make any public guarantees on scheduling, though, the internal documentation clearly states that internal tasks are preferred over external submissions.

I guess, you could say that this is a fairness issue that only matters in high load or overload scenarios anyway, so in that case giving advantage to local tasks is the best thing to do because this way you can do strictly less work than trying to steal work from the external queue in those cases. In consequence, in absence of any blocking (or busy-polling) on the pool (which is a bad idea anyway), this should not be a problem as long as the application can make some progress.

On the other hand, @viktorklang and @drewhk considered this to be a problem, so maybe I'm missing a critical scenario? In the meantime we added the internal dispatcher which should at least prevent that more critical internal workloads are completely starved. (Also OSs implement preemption for a reason you would think...)

[patriknw]

I agree with your reasoning and I think we can remove the test.

[drewhk]

My two cents: I generally freak out of any fairness issue because they are the silent killer. In fact, the problem is exactly that they come out usually in high-load scenarios and then instead of graceful degradation the system suddenly enters unpredictable failure modes that are very hard to diagnose and debug. If you have read the Roblox post-mortem, one of the main causes was a fairness issue.

In this case, if I recall correctly the original issue was cross-dispatcher submissions.

[drewhk]

This was the original ticket: #17341