RFC 0002: MT Execution Contexts

[ysbaddaden]

No description provided.

[beta-ziliani]

Preview: https://github.com/crystal-lang/rfcs/blob/rfc-0002-mt-execution-contexts/text/0002-execution-contexts.md

[straight-shoota]

The distinction between execution context and scheduler could need a bit refinement. There is definitely some overlap in functionality, just by comparing the API. I guess execution contexts might take over some features of the current scheduler?

[RX14]

Absolutely lovely, well-written proposal. I completely agree with the design intent here, and only have a few—mostly overlapping—comments about event loops and the default context. The vast majority of this design is exactly what I would like to see in crystal.

[RX14]

This proposal supports creating multiple execution contexts, let the application configure it's own EC and start fibers in it if required. That allows all the complexity the app needs when configuring the context the application actually runs in, because it's initialized by the application. The root context does not have to be well-used by the application.

[ysbaddaden]

@RX14 Tell me if I'm wrong, the plan would be:

Crystal 1

• introduce EC with ST and MT;
• deprecate same_thread argument;
• same_thread: false is NOOP;
• ST accepts same_thread: true (always true anyway);
• MT raises on same_thread: true (new API, no breaking change);
• default EC is ST (no breaking change);
• consider a -Dmt flag to force default EC to be MT (?);

Crystal 2

• remove deprecated same_thread (breaking change);
• default EC becomes MT (breaking change).

Isolated context

I think I see that context for UI loops only, and want to prevent blocking behaviors, but there's nothing wrong with doing blocking calls in other use cases, and using the event-loop normally is fine. Still, spawning a fiber without an explicit context should either raise or the default context should be configured (as you suggest):

abstract class ExecutionContext
  class Isolated < ExecutionContext
    def initialize(name : String, @spawn_context : ExecutionContext? = nil, &)
      @thread = Thread.new(name) { yield }
    end

    def spawn(**args, &) : Fiber
      if ctx = @spawn_context
        ctx.spawn(**args) { yield }
      else
        raise RuntimeError.new("Can't spawn in isolated context (need a spawn context)")
      end
    end
  end
end

mt = ExecutionContext::MultiThreaded.new
ui = ExecutionContext::Isolated.new("GTK", spawn_context: mt) { Gtk.main }

Instead of raising, the spawn context could be the default EC.

[ysbaddaden]

@RX14 I applied your suggestions to the RFC.

[RX14]

@ysbaddaden I think -Dmt is probably not necessary, and the exact implementation plan for crystal 2 is best left deferred until there's operational experience, but I agree on everything else.

I envision the root execution context being MT or ST a moot point, because every well-architected app has a single App.run line at the top-level and converting that top-level code to be spawning a MT context and waiting for that fiber should be a one-liner if we have the right helper methods in place.

If we all agree, maybe we can start on the other 90% of the RFC: bikeshedding naming. I like ExecutionContext::Parallel, because I don't like the idea of implementation details (threads) leaking into the name.

[ysbaddaden]

@RX14 The mt flag may not be necessary in Crystal v1, as the default context could be MT:1 and resized on demand (still no breaking change). I'll still push for MT:N to be the default in Crystal v2. Execution contexts are a mean to further control the parallelism in very specific cases, not the end solution. I believe developers shouldn't have to care about it until you have to.

I wouldn't bikeshed the namings just yet. As I'm experimenting with the types, I feel that the difference is getting thinner and thinner. In fact, Kotlin only has a single scheduler implementation, and a couple constructors to start execution contexts with 1 (ST) or many threads (MT).

I'm also struggling with the inheritance: EC::MT < EC makes sense, but so does EC::MT::Scheduler < EC as we want EC.current to point to the current MT scheduler running on the thread, not the shared MT context (it's easier to reach the context from the scheduler).

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/crystal-multithreading-support/6622/8

[ysbaddaden]

I forgot again but MT:1 would break spawn(same_thread: true) in Crystal 1. It's a NOOP without the preview_mt flag but the parameter was still exposed to the public API 😭

[straight-shoota]

I think we can accept breakage with same_thread: true. It only works with preview_mt which is explicitly a preview feature. There should be no expectations on compatibility in a setting outside of preview_mt.

[yxhuvud]

I like this, I like this a lot.

[yxhuvud]

I think data and especially execution locality could show up on the negative side as well, as the round robin take away a lot of programmatic control of data locality as well. It is .. possible.. to manually schedule fibers to dedicated threads but it really is not the way it currently is meant to be used.

[yxhuvud]

I'm in the 'let the event loop decide if it want to be instantiated on a thread, execution level or global level' camp. How that would look API-wise I'm less sure - especially not if dynamic amount of threads in a context is to be supported.

This might be complex: I don’t think we can share a libevent across event bases

From what I have gathered from libevent docs, it is possible, but would necessitate a lot more synchronization when io happens (*), so it is probably slower.

But yes, it is complex. Windows, and its weird file handles says hi. Each open file handle is specific for each instance of whatever it uses, so it needs to be only one global one event instance there.

• we already enable some structures for thread safety but then create separate bases for each thread anyhow, IIRC. It was quite a while since I looked at it. I think we can remove that enabling without danger - they should really only be used when actually reusing a libevent base between threads). We don't use the specialized mt safe functions libevent that make use of it.

[yxhuvud]

Allowing a dynamic amount of threads requires more synchronization and complexity than having a static amount. While it sounds nice to be able to adjust, it probably warrants its own separate class. Making certain all threads are in a waiting state before starting to actually queue stuff allows a bunch of simplifications with less mutexes and risks a lot fewer possible race conditions too.

[ysbaddaden]

Enqueue doesn't have to need much sync. Go pushes to a bounded local queue (per scheduler) with overflow to a global queue; threads can be started at any time: when they reach the run loop they will grab a batch of fibers from the global queue or steal from another scheduler. Stopping ain't more complex, schedulers aren't tied to a specific thread, the thread detaches the scheduler and returns itself to the thread pool.

The complexity is more in when to start / stop a thread.

[ysbaddaden]

And that complexity pushes for it to become a "future evolution".

[Blacksmoke16]

As someone who isn't familiar with this stuff at all, my random question is:

Do we need to do anything in relation to like how Intel has P and E cores now? Like as a way to signal to the OS's thread scheduler that a fiber should have a preference on where it runs? Or is that something the OS itself handles somehow?

[ysbaddaden]

@Blacksmoke16 From what I read specifying a thread priority can hint the OS to schedule the thread on a big (efficient) or little (power) core. We can also set a thread affinity to a given core, but we must detect the core type beforehand.

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/timeline-for-multithreading-support/3604/21

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/upcoming-release-1-14-0/7199/1

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/new-event-loop-unix-call-for-reviews-tests/7207/6

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/charting-the-route-to-multi-threading-support/7320/1

[Qard]

Just wanted to share some thoughts/ideas I'm using in my own Fiber-based multi-threaded language.

There are two distinct access patterns that can be isolated into different types of Fiber construction:

• In the typical case it is desirable that a Fiber takes access to variables present in the lexical scope in which it is constructed. As such, distributing to another thread creates synchronization challenges which make doing so undesirable.
• In some cases you have a singular "task" you want distributed which it would be reasonable that the processor takes ownership of that in the sense that a language like Rust does.

How I chose to handle this is with two distinct Fiber types: a local Fiber which can access any in-scope data but can only execute on the thread in which it was created, and an ownership-claiming Fiber which specifically avoids running on the thread in which it was created, taking ownership of a passed in value and bringing that over with it to whatever thread it gets dispatched to.

On top of that I have a work-stealing mechanism which can pull a Fiber back to its origin thread when it's not busy, but the Fiber would otherwise be guaranteed to run on a different thread.

This design focuses very strongly on maintaining locality to achieve the best performance and to avoid synchronization complexity as much as possible while making distribution over threads certain when loop utilization would benefit from it.

In a typical app server you're going to see http requests come in which are largely isolated from each other and can be immediately distributed to other threads, but then the processing within those tends to need to share more with each other, and what you need within the scope of parts of that request is more likely just concurrency and not actually parallelism. If you need parallelism you can reach for that tool, but if a more restrictive tool is provided to ensure actual isolation of the work for that Fiber then it can be much more effectively distributed.

Another thing you see often with http requests is that most object references within a request are to objects allocated in that request, but there are a small number of exceptions which typically fit neatly into a shared interface category such as a database connection (or pool) or an app server exposing utilities to each route. For these types I opted to build actors into my language and have any interaction with these objects go through a shim copied into the ownership-taking fibers which dispatch these interactions across threads through Fiber yields as the "lock" mechanism. A task simply gets passed over to the thread owning the object to do that interaction and the result of that interaction resolves the yield on the other end.

In my language I went for an explicit form of this actor type specialization, but you could also go for explicit construction of a proxy type when passing to an ownership-taking Fiber.

In any case, what I wanted to convey is that in my experience there are two distinct ways in which people want to distribute work with Fibers and isolating those two behaviours can actually have a lot of benefits. 🙂

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/upcoming-release-1-15-0/7537/1

[ysbaddaden]

The current constructors are rather complex and inconvenient to type, and pushing ExecutionContext under the Fiber namespace didn't help. I mean:

ctx = Fiber::ExecutionContext::Isolated.new(name) { }
ctx = Fiber::ExecutionContext::SingleThreaded.new(name)
ctx = Fiber::ExecutionContext::MultiThreaded.new(name, size: 1..4)

I'm wondering if we could have simpler helpers. For example (thinking out loud, don't take things for granted):

ctx = Fiber.start_isolated(name) { } # or #spawn_isolated ?
ctx = Fiber.start_concurrent_context(name)
ctx = Fiber.start_parallel_context(name, size: 1..4)

I'm not sure it's really better. Maybe a bit more explicit thanks to the start_ prefix?

The naming might also indicate that maybe the ST and MT classes should be named Concurrent and Parallel of SingleThreaded and MultiThreaded?

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/upcoming-release-1-16-0/7883/1

[ysbaddaden]

I updated the phrasing and names of the proposed execution contexts.

Reading the RFC again, I believe that its purpose, to propose a new API to achieve parallelism in Crystal, can be considered done, even though the feature isn't fully released.

I propose to archive RFC 0002 and to start a new one with details on the actual execution contexts (Concurrent, Parallel, Isolated), what guarantees they bring and how they differ, and what still needs to be done: resize, shutdown, thread pool and detach on syscall, mostly.

[straight-shoota]

polish: Maybe this wording is a bit more clear?

Suggested change

**Concurrent Context**: fibers will never run in parallel, they can use simpler and faster synchronization primitives internally (no atomics, no thread safety) and still communicate with other contexts with the default thread-safe primitives (e.g. `Channel`); the drawback is that a blocking fiber will block the other fibers from progressing. The concurrency limitation doesn't mean that the fibers will keep running on the same system thread forever.

**Concurrent Context**: fibers will never run in parallel, they can use simpler and faster synchronization primitives internally (no atomics, no thread safety) and still communicate with other contexts with the default thread-safe primitives (e.g. `Channel`); the drawback is that a blocking fiber will block the other fibers from progressing. This concurrency limitation however does not guarantee that fibers stay on the same system thread.

[straight-shoota]

issue: an is actually correct because MT is pronounced as em-tee.

[straight-shoota]

issue: There's actually no execution_context parameter. Some of the other implementations are also not accurate (e.g. ::sleep only delegates to Fiber::ExecutionContext.current.sleep). We should validate all of them.

Suggested change

	def spawn(*, name : String?, execution_context : Fiber::ExecutionContext = Fiber::ExecutionContext.current, &block) : Fiber
	execution_context.spawn(name: name, &block)
	end
	def spawn(*, name : String?, &block) : Fiber
	Fiber::ExecutionContext::Scheduler.current.spawn
	.spawn(name: name, &block)
	end

[straight-shoota]

thought: I'm wondering if this should be an enum instead of arbitrary strings?

[straight-shoota]

thought: I'm still not sure if it's even worth having a separate concurrent context when there's not much practical benefit over a parallel context with max=1.

[straight-shoota]

thought: I suppose it would probably be more effort and not worth it to implement something different.
But technically, we probably don't even need a separate fiber for the event loop. In an isolated context, at any point we can only wait on a single event at most. So that could potentially just be interweaved directly into the isolated fiber. Avoiding fiber swapping. Probably not worth it, just mentioning the possibility.

[straight-shoota]

suggestion: The codegen fibers should probably inform the waitgroup.

Suggested change

	ncpu.times do
	codegen.spawn do
	# (runs in the codegen context)
	while unit = channel.receive?
	unit.compile
	end
	end
	end
	wg.add ncpu
	ncpu.times do
	codegen.spawn do
	# (runs in the codegen context)
	while unit = channel.receive?
	unit.compile
	end
	wg.done
	end
	end

[straight-shoota]

thought: When we implement rescaling for parallel contexts, the default context can be parallel with max=1 but resizable from user code.

And I believe at least upscaling shouldn't be much more complicated than setting Parallel#capacity to a higher number and initialize the associated schedulers.

[straight-shoota]

suggestion: The summary should be dead simple and understandable without reading up on terms only defined in this document.

Suggested change

	Reinvent MT support in Crystal to be more efficient, for example by avoiding blocked fibers while there are sleeping threads, all the while empowering developers with different Execution Contexts to run fibers in.
	Reinvent Crystal's concurrency execution model with respect to multi-threading. It's getting more efficient, for example by avoiding blocked fibers while there are sleeping threads. And it empowers developers with different strategies for scheduling fiber, called _Execution Contexts_.

[straight-shoota]

suggestion: simplify:

Suggested change

	Applications can create any number of execution contexts in parallel. These contexts are isolated but they shall still be capable to communicate together with the usual synchronization primitives (e.g. Channel, Mutex) that must be thread-safe.
	Applications can create any number of execution contexts in parallel. These contexts are isolated but they shall still be capable to communicate together with the usual thread-safe synchronization primitives (e.g. `Channel`, `Mutex`).