diff --git a/404.html b/404.html
index a564f12d..35b8e724 100644
--- a/404.html
+++ b/404.html
@@ -18,7 +18,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/404.html">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <!--
 Made with Book Theme
 https://github.com/alex-shpak/hugo-book
diff --git a/categories/index.html b/categories/index.html
index 992c29e5..db45d147 100644
--- a/categories/index.html
+++ b/categories/index.html
@@ -17,7 +17,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/categories/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <link rel="alternate" type="application/rss+xml" href="https://tassiluca.github.io/direct-style-experiments/categories/index.xml" title="direct-style-experiments" />
 <!--
 Made with Book Theme
diff --git a/docs/01-overview/index.html b/docs/01-overview/index.html
index 73031544..2434a405 100644
--- a/docs/01-overview/index.html
+++ b/docs/01-overview/index.html
@@ -11,7 +11,7 @@
 <meta property="og:type" content="article" />
 <meta property="og:url" content="https://tassiluca.github.io/direct-style-experiments/docs/01-overview/" /><meta property="article:section" content="docs" />
 
-<meta property="article:modified_time" content="2024-03-07T12:58:54+01:00" />
+<meta property="article:modified_time" content="2024-03-07T13:31:46+01:00" />
 
 <title>01 Overview | direct-style-experiments</title>
 <link rel="manifest" href="/direct-style-experiments/manifest.json">
@@ -19,7 +19,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/docs/01-overview/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <!--
 Made with Book Theme
 https://github.com/alex-shpak/hugo-book
@@ -229,8 +229,8 @@ <h2 id="the-contribution">
 <p>The <a href="../04-channels">second</a> introduces the main communication and synchronization primitive of direct style, channels, and how they can be used to exchange data between concurrent tasks (either <code>Future</code>s or coroutines).</p>
 <p>In this context, a contribution has been made by proposing the extension of the Scala Gears library with the following two abstractions, currently missing, inspired by those of the Kotlin Coroutines:</p>
 <ul>
-<li><strong>terminable channels</strong>, i.e. a channel that can be terminated, but whose values can still be read by consumers after its termination until all values are consumed;</li>
-<li><strong><code>Flow</code>s</strong>, modeling a <em>cold</em> stream of asynchronously computed values which are particularly useful for implementing asynchronous functions that produce, not just a single, but a sequence of values.</li>
+<li><em>terminable channels</em>, i.e. a channel that can be terminated, but whose values can still be read by consumers after its termination until all values are consumed;</li>
+<li><em><code>Flow</code>s</em>, modeling a <em>cold</em> stream of asynchronously computed values which are particularly useful for implementing asynchronous functions that produce, not just a single, but a sequence of values.</li>
 </ul>
 <p>The <a href="../05-rears">last example</a> investigates how to implement a reactive-like event-based system using direct style, taking as a use case a small sensor control system in an IoT context that needs to react to events coming from different sensors.</p>
 <p>To accomplish this, since Scala Gears lacks any kind of transforming operator, some have been introduced on top of channels (taking cues from Reactive frameworks) allowing them to be manipulated functionally.</p>
@@ -241,11 +241,11 @@ <h2 id="conclusions">
 <p>The current implementation of monadic Futures present in the standard library is insufficient to handle modern asynchronous programming in a structured and safe way: it lacks structured concurrency and cancellation mechanism, is not referential transparent and requires to be ugly mixed with direct style constructs to be used appropriately.</p>
 <p>This leads to the adoption of effectful libraries that offer these and many other powerful abstractions, beautifully wrapped in monadic constructs.
 This is the main pro and con of the monadic approach: what makes monads remarkable is their capability to turn statements into programmable values and introduce constructs to transform and compose them functionally in a very elegant (and somehow &ldquo;magical&rdquo; for ones who are not familiar with them) way.
-Despite this idyllic beauty, monads and effectful libraries require relevant expertise in functional programming to be fully grasped and composed effectively.</p>
+Despite this idyllic beauty, monads and effectful libraries require relevant expertise in functional programming to be fully grasped and effectively composed.</p>
 <p>Direct style frameworks are indeed arising as a more natural and intuitive way to handle concurrency, leveraging an imperative-like programming style that is familiar to all developers.</p>
 <p>In the JVM ecosystem, the most adopted and known direct-style library is the Kotlin Coroutines, which were introduced in 2018 by modifying the Kotlin language (rather than its runtime, like the project Loom has recently done with Virtual Thread) to support suspending functions.
 The main advantages of Kotlin Coroutines are that they provide suspension and cancellation mechanisms that are simple to understand and use, as well as a good ecosystem for channels and <code>Flow</code>s.
-Despite this, Coroutines are not still perfect: due to their design they partially suffer from the <a href="https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/">colored functions problem</a> and we need to be aware we can not use the same synchronization concurrency abstractions that we would use in Java with threads (like locks, <code>synchronized</code>, &hellip;) cause they are not designed to be used in the coroutines context.</p>
+Despite this, Coroutines are not still perfect: due to their design, they partially suffer from the <a href="https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/">colored functions problem</a> and we need to be aware we can not use the same synchronization concurrency abstractions that we would use in Java with threads (like locks, <code>synchronized</code>, &hellip;) cause they are not designed to be used in the coroutines context.</p>
 <p>Scala Gears is an attempt to bring direct style into the Scala ecosystem.
 Its API design is inspired by Kotlin Coroutines and, despite the fact it achieves suspension, unlike Kotlin, leveraging Virtual Threads for JVM and delimited continuation for Scala Native, the majority of constructs can be mapped to the Kotlin Coroutines ones.
 Despite being a very young project, it already offers a good set of abstractions for asynchronous programming, although it cannot yet be considered a mature library ready to be used in a production environment:</p>
@@ -253,10 +253,10 @@ <h2 id="conclusions">
 <li>some design choices should be addressed:
 <ul>
 <li>closing a channel prevents any further reading, precluding the possibility of processing the remaining values (see <a href="../04-channels">second</a> example);</li>
-<li>Task scheduling behavior has a strange behavior with higher-order functions (see <a href="../05-rears">third</a> example);</li>
+<li>Task scheduling behaves differently with higher-order functions depending on its signature and wither or not the function passed is suspendable (see <a href="../05-rears">third</a> example);</li>
 </ul>
 </li>
-<li>missing abstractions: the library is still missing some important abstractions, like the proposed <code>Flow</code>s for handling a cold stream of asynchronously computed values and operators for functionally transforming them;</li>
+<li>the library is still missing some important abstractions, like the proposed <code>Flow</code> for handling a cold stream of asynchronously computed values, and operators for functionally transforming channels (and in a next future, hopefully, <code>Flow</code>s or equivalent abstraction);</li>
 <li>performances: the project has been created for experimenting, thus performances have not been considered a priority so far, <a href="https://github.com/lampepfl/gears/blob/main/docs/summary-2023-06.md#performance">even though a comparison in overheads of
 the core primitives has been published</a>.</li>
 </ul>
@@ -309,7 +309,7 @@ <h2 id="references">
   <div class="flex flex-wrap justify-between">
 
 
-  <div><a class="flex align-center" href="https://github.com/tassiLuca/direct-style-experiments/commit/e61044e3afd86718ea14d14cd1442f44e99a713e" title='Last modified by Luca Tassinari | March 7, 2024' target="_blank" rel="noopener">
+  <div><a class="flex align-center" href="https://github.com/tassiLuca/direct-style-experiments/commit/38b3128e4598ddef38be0e451a3c02576e4eb470" title='Last modified by Luca Tassinari | March 7, 2024' target="_blank" rel="noopener">
       <img src="/direct-style-experiments/svg/calendar.svg" class="book-icon" alt="Calendar" />
       <span>March 7, 2024</span>
     </a>
diff --git a/docs/02-boundaries/index.html b/docs/02-boundaries/index.html
index d5148863..be020979 100644
--- a/docs/02-boundaries/index.html
+++ b/docs/02-boundaries/index.html
@@ -21,7 +21,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/docs/02-boundaries/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <!--
 Made with Book Theme
 https://github.com/alex-shpak/hugo-book
diff --git a/docs/03-basics/index.html b/docs/03-basics/index.html
index f011db9d..008dfe23 100644
--- a/docs/03-basics/index.html
+++ b/docs/03-basics/index.html
@@ -21,7 +21,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/docs/03-basics/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <!--
 Made with Book Theme
 https://github.com/alex-shpak/hugo-book
diff --git a/docs/04-channels/index.html b/docs/04-channels/index.html
index 07ff087f..30d7b512 100644
--- a/docs/04-channels/index.html
+++ b/docs/04-channels/index.html
@@ -19,7 +19,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/docs/04-channels/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <!--
 Made with Book Theme
 https://github.com/alex-shpak/hugo-book
diff --git a/docs/05-rears/index.html b/docs/05-rears/index.html
index c285fba1..f6467fcb 100644
--- a/docs/05-rears/index.html
+++ b/docs/05-rears/index.html
@@ -21,7 +21,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/docs/05-rears/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <!--
 Made with Book Theme
 https://github.com/alex-shpak/hugo-book
diff --git a/docs/index.html b/docs/index.html
index f3eb9478..c142198b 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -17,7 +17,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/docs/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <link rel="alternate" type="application/rss+xml" href="https://tassiluca.github.io/direct-style-experiments/docs/index.xml" title="direct-style-experiments" />
 <!--
 Made with Book Theme
diff --git a/en.search-data.min.5cd36aacfc646802f6043b59de31ec5fe9b0983544fd189d576e27dcc8106546.json b/en.search-data.min.5cd36aacfc646802f6043b59de31ec5fe9b0983544fd189d576e27dcc8106546.json
new file mode 100644
index 00000000..eb6fa004
--- /dev/null
+++ b/en.search-data.min.5cd36aacfc646802f6043b59de31ec5fe9b0983544fd189d576e27dcc8106546.json
@@ -0,0 +1 @@
+[{"id":0,"href":"/direct-style-experiments/docs/01-overview/","title":"01 Overview","section":"Docs","content":" Overview of the project # Context # In the realm of asynchronous programming, the Scala ecosystem offers a set of solid and widely adopted monadic constructs and libraries to tackle complex tasks functionally with elegance and efficiency, like Monix Tasks and Cats Effecs, enabling a wide range of interesting and useful features, like composable error handling, cancellation mechanisms and structured concurrency that the standard library lacks. However, they also come with a cost: the pervasiveness of the flatMap operator to compose values makes the code harder to reason about and difficult and awkward to integrate with regular control structures.\nIn the last years, we have been assisting the increase in adoption of continuation and coroutines in modern runtimes, either exploiting some kind of fibers support, like the project Loom with Virtual Threads, or via code generation, like Kotlin Coroutines, aiming to capture the essence of effects more cleanly compared to monads.\nGoals # The goal of this project is to explore, mainly focusing on Scala, the direct style, developing a few examples (not too complex) leveraging the new strawman library Gears, comparing it with Kotlin Coroutines and the current implementation of monadic Futures, seeking to analyze aspects such as:\nergonomics of the two styles; which of the two approaches has a real advantage in adoption; pros and cons of the two styles; any limitations and difficulties encountered in using them. The contribution # The project is built around three small examples.\nThe first aims, through the implementation of the core of a small Web service, to introduce the basics of asynchronous programming in direct style, focusing on structured concurrency and cancellation.\nThe second introduces the main communication and synchronization primitive of direct style, channels, and how they can be used to exchange data between concurrent tasks (either Futures or coroutines).\nIn this context, a contribution has been made by proposing the extension of the Scala Gears library with the following two abstractions, currently missing, inspired by those of the Kotlin Coroutines:\nterminable channels, i.e. a channel that can be terminated, but whose values can still be read by consumers after its termination until all values are consumed; Flows, modeling a cold stream of asynchronously computed values which are particularly useful for implementing asynchronous functions that produce, not just a single, but a sequence of values. The last example investigates how to implement a reactive-like event-based system using direct style, taking as a use case a small sensor control system in an IoT context that needs to react to events coming from different sensors.\nTo accomplish this, since Scala Gears lacks any kind of transforming operator, some have been introduced on top of channels (taking cues from Reactive frameworks) allowing them to be manipulated functionally.\nConclusions # The current implementation of monadic Futures present in the standard library is insufficient to handle modern asynchronous programming in a structured and safe way: it lacks structured concurrency and cancellation mechanism, is not referential transparent and requires to be ugly mixed with direct style constructs to be used appropriately.\nThis leads to the adoption of effectful libraries that offer these and many other powerful abstractions, beautifully wrapped in monadic constructs. This is the main pro and con of the monadic approach: what makes monads remarkable is their capability to turn statements into programmable values and introduce constructs to transform and compose them functionally in a very elegant (and somehow \u0026ldquo;magical\u0026rdquo; for ones who are not familiar with them) way. Despite this idyllic beauty, monads and effectful libraries require relevant expertise in functional programming to be fully grasped and effectively composed.\nDirect style frameworks are indeed arising as a more natural and intuitive way to handle concurrency, leveraging an imperative-like programming style that is familiar to all developers.\nIn the JVM ecosystem, the most adopted and known direct-style library is the Kotlin Coroutines, which were introduced in 2018 by modifying the Kotlin language (rather than its runtime, like the project Loom has recently done with Virtual Thread) to support suspending functions. The main advantages of Kotlin Coroutines are that they provide suspension and cancellation mechanisms that are simple to understand and use, as well as a good ecosystem for channels and Flows. Despite this, Coroutines are not still perfect: due to their design, they partially suffer from the colored functions problem and we need to be aware we can not use the same synchronization concurrency abstractions that we would use in Java with threads (like locks, synchronized, \u0026hellip;) cause they are not designed to be used in the coroutines context.\nScala Gears is an attempt to bring direct style into the Scala ecosystem. Its API design is inspired by Kotlin Coroutines and, despite the fact it achieves suspension, unlike Kotlin, leveraging Virtual Threads for JVM and delimited continuation for Scala Native, the majority of constructs can be mapped to the Kotlin Coroutines ones. Despite being a very young project, it already offers a good set of abstractions for asynchronous programming, although it cannot yet be considered a mature library ready to be used in a production environment:\nsome design choices should be addressed: closing a channel prevents any further reading, precluding the possibility of processing the remaining values (see second example); Task scheduling behaves differently with higher-order functions depending on its signature and wither or not the function passed is suspendable (see third example); the library is still missing some important abstractions, like the proposed Flow for handling a cold stream of asynchronously computed values, and operators for functionally transforming channels (and in a next future, hopefully, Flows or equivalent abstraction); performances: the project has been created for experimenting, thus performances have not been considered a priority so far, even though a comparison in overheads of the core primitives has been published. In conclusion, Scala Gears is a promising project that could bring direct-style async programming into the Scala ecosystem, giving, together with boundary and break Scala 3 support, a nice alternative to the current monadic approach, simplifying the way we handle concurrency, making it more natural and intuitive.\nReferences # Scala Gears, Programming Methods Laboratory EPFL Scala 3: What Is \u0026ldquo;Direct Style\u0026rdquo; by D. Wampler Kotlin Coroutines documentation Pre-SIP: Suspended functions and continuations in Scala 3 The Great Concurrency Smackdown: ZIO versus JDK by John A. De Goes Continuaton, coroutine, and generator by A. Ber KotlinConf 2017 - Introduction to Coroutines by Roman Elizarov KotlinConf 2017 - Deep Dive into Coroutines on JVM by Roman Elizarov Kotlin Co-routine Scope by A. Nadiger What Color is your function by B. Nystrom Channel in Kotlin Coroutines SharedFlow and StateFlow Demystifying Kotlin\u0026rsquo;s Channel Flows by S. Cooper Home Next: boundary \u0026amp; break "},{"id":1,"href":"/direct-style-experiments/docs/02-boundaries/","title":"02 Boundaries","section":"Docs","content":" boundary \u0026amp; break # boundary \u0026amp; break Modeling error handling data types with non-local breaks Optional Either + ? boundary \u0026amp; break mechanism provides a cleaner alternative to non-local returns:\nboundary: is short for boundary.apply: the indented code below, passed as body, is a context function that is called within boundary.apply to break, an in-scope given instance of Label is required (i.e. is impossible to break without an enclosing boundary) Users don\u0026rsquo;t define Label instances themselves. Instead, this is done inside the implementation of boundary.apply to provide the capability of doing a non-local return [Ref]: // From the Scala 3 standard library /** Run `body` with freshly generated label as implicit argument. * Catch any breaks associated with that label and return their * results instead of `body`\u0026#39;s result. */ inline def apply[T](inline body: Label[T] ?=\u0026gt; T): T = val local = Label[T]() try body(using local) catch case ex: Break[T] @unchecked =\u0026gt; if ex.label eq local then ex.value else throw ex non-local breaks are implemented as non-fatal exceptions: the implementation is optimized to suppress unnecessary stack traces (which makes exceptions very slow); stack traces are useless since the exceptions are managed rather than exposed to the user abruptly enhanced performance is achieved when a break occurs within the same method, allowing it to be rewritten as a jump call to the enclosing scope within the same stack frame. boundary and break can be particularly useful for error handling (later examples will show some use cases) and inner loops where we need a short exit path. But, most importantly, they lay the foundations (along with a resume mechanism) for building new direct-style concurrency abstractions based on suspensions. Modeling error handling data types with non-local breaks # [Here you can find the full source.]\nIn the following section are presented two data types that can be used to handle errors, both leveraging the boundary and break mechanism. The first (optional) has been presented in the Scalar conference by M. Odersky, while the second has been implemented to apply the same style also to Either data type.\nOptional # /** Represents a computation that will hopefully return * [[Some]]thing or simply [[None]] if it can\u0026#39;t. */ object optional: /** Defines the boundary for an [[Option]] returning computation, * whose [[body]] is given in input. */ inline def apply[T](inline body: Label[None.type] ?=\u0026gt; T): Option[T] = boundary(Some(body)) extension [T](o: Option[T]) /** @return the enclosed [[Option]] object if defined, or break * to the enclosing boundary with [[None]]. */ inline def ?(using label: Label[None.type]): T = o.getOrElse(break(None)) Either + ? # /** Represents a computation that will hopefully return a [[Right]] value but might fail with a [[Left]] one.*/ object either: /** Defines the boundary for the [[Either]] returning computation, whose [[body]] is given in input. */ inline def apply[L, R](inline body: Label[Left[L, Nothing]] ?=\u0026gt; R): Either[L, R] = boundary(Right(body)) /** Quickly break to the enclosing boundary with a [[Left]] filled with [[l]]. */ inline def left[L, R](l: L)(using Label[Left[L, Nothing]]): Either[L, R] = break(Left(l)) extension [L, R](e: Either[L, R]) /** @return this [[Right]] value or break to the enclosing boundary with the [[Left]] value. */ inline def ?(using Label[Left[L, Nothing]]): R = e match case Right(value) =\u0026gt; value case Left(value) =\u0026gt; break(Left(value)) extension [R](t: Try[R]) /** @return this [[Success]] value or break to the enclosing boundary with a [[Left]] containing * the converted `Throwable` exception performed by the implicit [[converter]]. */ inline def ?[L](using Label[Left[L, Nothing]])(using converter: Conversion[Throwable, L]): R = t match case Success(value) =\u0026gt; value case Failure(exception) =\u0026gt; break(Left(converter(exception))) /** An object encapsulating a collection of `Throwable` given converters. */ object EitherConversions: /** Converts a `Throwable` to a `String` with its message. */ given Conversion[Throwable, String] = _.getMessage This kind of data type will be particularly useful in the next examples to quickly break in case of failures, returning the caller a meaningful error message, and simplifying the error-handling code.\nOverview Next: Basic asynchronous constructs "},{"id":2,"href":"/direct-style-experiments/docs/03-basics/","title":"03 Basics","section":"Docs","content":" Basic asynchronous constructs # Basic asynchronous constructs The need for a new Future construct Example: a blog posts service Structure Current monadic Future Direct style: Scala version with gears Kotlin Coroutines Takeaways The need for a new Future construct # The current implementation of the Future monadic construct suffers the following main cons:\nLack of referential transparency; Lack of cancellation mechanisms and structured concurrency; Accidental Sequentiality. To show these weaknesses in practice, a simple example of the core of a web service implementation is presented.\nExample: a blog posts service # Idea: develop a very simple (mocked) service that allows retrieving and storing from a repository blog posts, performing checks on the content and author before the actual storage. The example has been implemented using:\nthe continuation style through the current Scala Future monadic constructs; the direct style, through: the abstractions offered by Gears; Kotlin coroutines. The example is organized into Gradle submodules:\nblog-ws-commons contains code that has been reused for both the monadic and direct versions; blog-ws-monadic contains the monadic Scala style; blog-ws-direct contains the direct version using Scala Gears; blog-ws-direct-kt contains the direct version using Kotlin Coroutines. For this example just the tests are provided. You can explore them in the test folders and run via Gradle using the name of the submodule:\n./gradlew :blog-ws-\u0026lt;monadic | direct | direct-kt\u0026gt;:test Structure # The domain is modeled using abstract data types in a common PostsModel trait:\n/** The model of a simple blog posts service. */ trait PostsModel: /** The post author\u0026#39;s identifier. */ type AuthorId /** The posts title. */ type Title /** The posts body. */ type Body /** The content of the post. */ type PostContent = (Title, Body) /** A post author and their info. */ case class Author(authorId: AuthorId, name: String, surname: String) /** A blog post, comprising an author, title, body and the last modification. */ case class Post(author: Author, title: Title, body: Body, lastModification: Date) /** A function that verifies the content of the post, returning [[Right]] with the content of * the post if the verification succeeds or [[Left]] with the reason why failed. */ type ContentVerifier = (Title, Body) =\u0026gt; Either[String, PostContent] /** A function that verifies the author has appropriate permissions, returning [[Right]] * with their information or [[Left]] with the reason why failed. */ type AuthorsVerifier = AuthorId =\u0026gt; Either[String, Author] To implement the service two components have been conceived, following the Cake Pattern:\nPostsRepositoryComponent exposes the Repository trait allowing to store and retrieve blog posts; mocks a DB technology with an in-memory collection. PostsServiceComponent is the component exposing the Service interface. it could be called by the controller of the ReSTful web service. Both must be designed in an async way.\nCurrent monadic Future # The interface of the repository and services component of the monadic version are presented hereafter and their complete implementation is available here.\n/** The component exposing blog posts repositories. */ trait PostsRepositoryComponent: context: PostsModel =\u0026gt; /** The repository instance. */ val repository: PostsRepository /** The repository in charge of storing and retrieving blog posts. */ trait PostsRepository: /** Save the given [[post]]. */ def save(post: Post)(using ExecutionContext): Future[Post] /** @return a [[Future]] completed with true if a post exists with * the given title, false otherwise. */ def exists(postTitle: Title)(using ExecutionContext): Future[Boolean] /** @return a [[Future]] completed either with a defined optional * post with given [[postTitle]] or an empty one. */ def load(postTitle: Title)(using ExecutionContext): Future[Option[Post]] /** Load the post with the given [[postTitle]]. */ def loadAll()(using ExecutionContext): Future[LazyList[Post]] /** The component blog posts service. */ trait PostsServiceComponent: context: PostsRepositoryComponent \u0026amp; PostsModel =\u0026gt; /** The blog post service instance. */ val service: PostsService /** The service exposing a set of functionalities to interact with blog posts. */ trait PostsService: /** Creates a new blog post with the given [[title]] and [[body]], authored by [[authorId]]. */ def create(authorId: AuthorId, title: Title, body: Body)(using ExecutionContext): Future[Post] /** Get a post from its [[title]]. */ def get(title: Title)(using ExecutionContext): Future[Option[Post]] /** Gets all the stored blog posts in a lazy manner. */ def all()(using ExecutionContext): Future[LazyList[Post]] All the exposed functions, since they are asynchronous, return an instance of Future[T] and require to be called in a scope where a given instance of the ExecutionContext is declared.\nWhat\u0026rsquo;s important to delve into is the implementation of the service, and, more precisely, of the create method. As already mentioned, before saving the post two checks need to be performed:\nthe post author must have permission to publish a post and their information needs to be retrieved (supposing they are managed by another service); the content of the post is analyzed in order to prevent the storage and publication of inappropriate content. Since these operations are independent from each other they can be spawned and run in parallel.\noverride def create(authorId: AuthorId, title: Title, body: Body)(using ExecutionContext): Future[Post] = for exists \u0026lt;- context.repository.exists(title) if !exists post \u0026lt;- save(authorId, title, body) yield post private def save(authorId: AuthorId, title: Title, body: Body)(using ExecutionContext): Future[Post] = val authorAsync = authorBy(authorId) val contentAsync = verifyContent(title, body) for content \u0026lt;- contentAsync author \u0026lt;- authorAsync post = Post(author, content._1, content._2, Date()) _ \u0026lt;- context.repository.save(post) yield post /* Pretending to make a call to the Authorship Service that keeps track of authorized authors. */ private def authorBy(id: AuthorId)(using ExecutionContext): Future[Author] = ??? /* Some local computation that verifies the content of the post is appropriate. */ private def verifyContent(title: Title, body: Body)(using ExecutionContext): Future[PostContent] = ??? This implementation shows the limits of the current monadic Future mechanism:\nif we want to achieve the serialization of futures execution we need to compose them using the flatMap, like in the create function: first, the check on the post existence is performed, and only if it is successful and another post with same title doesn\u0026rsquo;t exist the save function is started\nas a consequence, if we want two futures to run in parallel we have to spawn them before the for-yield, as in the save function. This is error-prone and could lead to unexpected sequentiality, like this:\n// THIS IS WRONG: the two futures are started sequentially! for content \u0026lt;- verifyContent(title, body) author \u0026lt;- authorBy(authorId) post = Post(author, content._1, content._2, Date()) _ \u0026lt;- context.repository.save(post) yield post since the publication of a post can be performed only if both of these checks succeed, it is desirable that, whenever one of the two fails, the other gets canceled. Unfortunately, currently, Scala Futures are not cancellable and provide no structured concurrency mechanism.\nmoreover, they lack referential transparency, i.e. future starts running when they are defined. This means that passing a reference to a future is not the same as passing the referenced expression.\nDirect style: Scala version with gears # The API of the gears library is presented hereafter and is built on top of four main abstractions, three of which are here presented (the fourth in the next example):\nAsync context is \u0026ldquo;a capability that allows a computation to suspend while waiting for the result of an async source\u0026rdquo;. Code that has access to an instance of the Async trait is said to be in an async context and can suspend its execution. Usually, it is provided via given instances. A common way to obtain an Async instance is to use an Async.blocking. Async.Source model an asynchronous source of data that can be polled or awaited by suspending the computation, as well as composed using combinator functions. Futures are the primary (in fact, the only) active elements that encapsulate a control flow that, eventually, will deliver a result (either a computed or a failure value that contains an exception). Since Futures are Async.Sources can be awaited and combined with other Futures, suspending their execution. Tasks are the abstraction created to create delayed Futures, responding to the lack of referential transparency problem. They take the body of a Future as an argument; its run method converts that body to a Future, starting its execution. Promises allow us to define Future\u0026rsquo;s result value externally, instead of executing a specific body of code. classDiagram class Async { \u003c\u003c trait \u003e\u003e +group: CompletionGroup +withGroup(group: CompletionGroup) Async +await[T](src: Async.Source[T]) T +current() Async$ +blocking[T](body: Async ?=\u003e T) T$ +group[T](body: Async ?=\u003e T) T$ } class `Async.Source[+T]` { \u003c\u003c trait \u003e\u003e +poll(k: Listener[T]) Boolean +poll() Option[T] +onComplete(k: Listener[T]) +dropListener(k: Listener[T]) +awaitResult()(using Async) T } Async *--\u003e `Async.Source[+T]` class OriginalSource { \u003c\u003c abstract class \u003e\u003e } `Async.Source[+T]` \u003c|-- OriginalSource class `Listener[-T]` { \u003c\u003c trait \u003e\u003e +lock: Listener.ListenerLock | Null +complete(data: T, source: Async.Source[T]) +completeNow(data: T, source: Async.Source[T]) Boolean +apply[T](consumer: (T, Source[T]) =\u003e Unit) Listener[T]$ } `Async.Source[+T]` *--\u003e `Listener[-T]` class `Future[+T]` { \u003c\u003c trait \u003e\u003e +apply[T](body: Async ?=\u003e T) Future[T]$ +now[T](result: Try[T]) Future[T] +zip[U](f2: Future[U]) Future[T, U] +alt(f2: Future[T]) Future[T] +altWithCancel(f2: Future[T]) Future[T] } class `Promise[+T]` { \u003c\u003c trait \u003e\u003e +apply() Promise[T]$ +asFuture Future[T] +complete(result: Try[T]) } OriginalSource \u003c|-- `Future[+T]` `Future[+T]` \u003c|-- `Promise[+T]` class `Task[+T]` { +apply(body: (Async, AsyncOperations) ?=\u003e T) Task[T]$ +run(using Async, AsyncOperations) Future[+T] } `Future[+T]` \u003c--* `Task[+T]` class Cancellable { \u003c\u003c trait \u003e\u003e +group: CompletionGroup +cancel() +link(group: CompletionGroup) +unlink() } Cancellable \u003c|-- `Future[+T]` class Tracking { \u003c\u003c trait \u003e\u003e +isCancelled Boolean } Cancellable \u003c|-- Tracking class CompletionGroup { +add(member: Cancellable) +drop(member: Cancellable) } Tracking \u003c|-- CompletionGroup Async *--\u003e CompletionGroup Going back to our example, the interface of both the repository and service components becomes (here you can find the complete sources):\n/** The component exposing blog posts repositories. */ trait PostsRepositoryComponent: context: PostsModel =\u0026gt; /** The repository instance. */ val repository: PostsRepository /** The repository in charge of storing and retrieving blog posts. */ trait PostsRepository: /** Save the given [[post]]. */ def save(post: Post)(using Async): Either[String, Post] /** Return true if a post exists with the given title, false otherwise. */ def exists(postTitle: Title)(using Async): Either[String, Boolean] /** Load the post with the given [[postTitle]]. */ def load(postTitle: Title)(using Async): Either[String, Option[Post]] /** Load all the saved post. */ def loadAll()(using Async): Either[String, LazyList[Post]] /** The blog posts service component. */ trait PostsServiceComponent: context: PostsRepositoryComponent \u0026amp; PostsModel =\u0026gt; /** The blog post service instance. */ val service: PostsService /** The service exposing a set of functionalities to interact with blog posts. */ trait PostsService: /** Creates a new blog post with the given [[title]] and [[body]], authored by [[authorId]], * or a string explaining the reason of the failure. */ def create(authorId: AuthorId, title: Title, body: Body)(using Async): Either[String, Post] /** Get a post from its [[title]] or a string explaining the reason of the failure. */ def get(title: Title)(using Async): Either[String, Option[Post]] /** Gets all the stored blog posts in a lazy manner or a string explaining the reason of the failure. */ def all()(using Async): Either[String, LazyList[Post]] As you can see, Futures are gone and the return type it\u0026rsquo;s just the result of their intent (expressed with Either to return a meaningful message in case of failure). The fact they are suspendable is expressed using the Async context, which is required to invoke those functions.\nKey inspiring principle (actually, taken by Kotlin)\n❝Concurrency is hard! Concurrency has to be explicit!❞\nBy default the code is serial. If you want to opt-in concurrency you have to explicitly use a Future or Task spawning a new control flow that executes asynchronously, allowing the caller to continue its execution.\nThe other important key feature of the library is the support for structured concurrency and cancellation mechanisms:\nFutures are Cancellable instances;\nWhen you cancel a future using the cancel() method, it promptly sets its value to Failure(CancellationException). Additionally, if it\u0026rsquo;s a runnable future, the thread associated with it is interrupted using Thread.interrupt().\nto avoid immediate cancellation, deferring the cancellation after some block, is possible using uninterruptible function:\nval f = Future: // this can be interrupted uninterruptible: // this cannot be interrupted *immediately* // this can be interrupted Futures are nestable; the lifetime of nested computations is contained within the lifetime of enclosing ones. This is achieved using CompletionGroups, which are cancellable objects themselves and serve as containers for other cancellable objects; once they are canceled, all of their members are canceled as well.\nThe group is accessible through Async.current.group; A cancellable object can be included inside the cancellation group of the async context using the link method; this is what the implementation of the Future does, under the hood. The implementation of the create function with direct style in Gears looks like this:\noverride def create(authorId: AuthorId, title: Title, body: Body)(using Async): Either[String, Post] = either: if context.repository.exists(title).? then left(s\u0026#34;A post entitled $title already exists\u0026#34;) val f = Future: val content = verifyContent(title, body).run val author = authorBy(authorId).run content.zip(author).await val (post, author) = f.awaitResult.? context.repository.save(Post(author.?, post.?._1, post.?._2, Date())).? /* Pretending to make a call to the Authorship Service that keeps track of authorized authors. */ private def authorBy(id: AuthorId): Task[Either[String, Author]] = ??? /* Some local computation that verifies the content of the post is appropriate. */ private def verifyContent(title: Title, body: Body): Task[Either[String, PostContent]] = ??? Some remarks:\nthe either boundary have been used to quickly return a Right[String, Post] object in case something goes wrong; authorBy and verifyContent returns referential transparent Task instances. Running them spawns a new Future instance; Thanks to structured concurrency and zip combinator we can obtain that if one of the nested two futures fails the enclosing future is cancelled, cancelling also all its unterminated children zip: combinator function returning a pair with the results if both Futures succeed, otherwise fail with the failure that was returned first.\nBe aware of the fact to achieve cancellation is necessary to enclose both the content verification and authorization task inside an enclosing Future, since the zip doesn\u0026rsquo;t provide a cancellation mechanism per se. The following code wouldn\u0026rsquo;t work as expected!\n// WRONG: doesn\u0026#39;t provide cancellation! val contentVerifier = verifyContent(title, body).run val authorizer = authorBy(authorId).run val (post, author) = contentVerifier.zip(authorizer).awaitResult.? 👉🏻 To showcase the structured concurrency and cancellation mechanisms of Scala Gears tests have been prepared:\nStructuredConcurrencyTest CancellationTest Other combinator methods, available on Futures instance:\nCombinator Goal Future[T].zip(Future[U]) Parallel composition of two futures. If both futures succeed, succeed with their values in a pair. Otherwise, fail with the failure that was returned first Future[T].alt(Future[T]) / Seq[Future[T]].altAll Alternative parallel composition. If either task succeeds, succeed with the success that was returned first. Otherwise, fail with the failure that was returned last (race all futures). Future[T].altWithCancel(Future[T]) / Seq[Future[T]].altAllWithCancel Like alt but the slower future is cancelled. Seq[Future[T]].awaitAll .await for all futures in the sequence, returns the results in a sequence, or throws if any futures fail. Seq[Future[T]].awaitAllOrCancel Like awaitAll, but cancels all futures as soon as one of them fails. Kotlin Coroutines # A coroutine in Kotlin is an instance of a suspendable computation.\nTheir API is quite similar to the Scala Gears, which has taken inspiration from Kotlin coroutines. To try to make a comparison, the following table shows the correspondence between the two libraries:\nScala Gears Kotlin Coroutines Async CoroutineScope Future[Unit] Job Future[T] Deferred\u0026lt;T\u0026gt; def all()(using Async) suspend fun all() In the Kotlin Coroutine library, CoroutineScope is an interface that defines a single property, coroutineContext, which returns the CoroutineContext that defines the scope in which the coroutine runs.\npublic interface CoroutineScope { /** Returns the context of this scope. */ public val coroutineContext: CoroutineContext } Every coroutine must be executed in a coroutine context, which is a collection of key-value pairs that provide contextual information for the coroutine, including a dispatcher, that determines what thread or threads the coroutine uses for its execution, and the Job of the coroutine, which represents a cancellable background piece of work with a life cycle that culminates in its completion.\nDifferent ways to create a scope:\nGlobalScope.launch launching a new coroutine in the global scope \u0026ndash; discouraged because it can lead to memory leaks; CoroutineScope(Dispatchers.Default), using the constructor with a dispatcher; runBlocking - equivalent to the Gears Async.blocking - provides a way to run a coroutine in the MainScope, i.e. on the main/UI thread. Useful dispatchers:\nDefault dispatcher: to run CPU-intensive functions. If we forget to choose our dispatcher, this dispatcher will be selected by default; IO dispatcher: to run I-O bound computation, where we block waiting for input-output operations to complete, like network-related operations, file operations, etc.; Unconfined dispatcher: it isn\u0026rsquo;t restricted to a particular thread, i.e. doesn\u0026rsquo;t change the thread of the coroutine, it operates on the same thread where it was initiated; Main dispatcher: used when we want to interact with the UI. It is restricted to the main thread. Several coroutine builders exist, like launch, async, withContext which accept an optional CoroutineContext parameter that can be used to specify the dispatcher and other context elements.\nfun main(): Unit = runBlocking { // this: CoroutineScope val job: Job = this.launch(Dispatchers.IO) { // launch a new coroutine and continue delay(1000L) // non-blocking delay for 1 second (default time unit is ms) print(\u0026#34;Kotlin Coroutine!\u0026#34;) // print after delay } val job2: Deferred\u0026lt;String\u0026gt; = async { delay(100L) \u0026#34;Hello\u0026#34; } print(job2.await() + \u0026#34; \u0026#34;) // wait until child coroutine completes job.join() // wait until the job is done and \u0026#34;Kotlin Coroutine!\u0026#34; is printed } suspending functions are marked with the suspend keyword; they can use other suspending functions to suspend the execution of a coroutine.\nCoroutines follow the principle of structured concurrency: coroutines can be arranged into parent-child hierarchies where the cancellation of a parent leads to the immediate cancellation of all its children recursively. Failure of a child with an exception immediately cancels its parent and, consequently, all its other children.\nGoing back to our example, the interface of the service with Kotlin coroutines looks like this (here you can find the complete sources):\n/** The service exposing a set of functionalities to interact with blog posts. */ interface PostsService { /** Creates a new post. */ suspend fun create(authorId: String, title: String, body: String): Result\u0026lt;Post\u0026gt; /** Retrieves a post by its title. */ suspend fun get(title: String): Result\u0026lt;Post\u0026gt; /** Retrieves all the posts. */ suspend fun getAll(): Result\u0026lt;Sequence\u0026lt;Post\u0026gt;\u0026gt; } The implementation of the create function:\noverride suspend fun create(authorId: String, title: String, body: String): Result\u0026lt;Post\u0026gt; = runCatching { coroutineScope { require(!repository.exists(title)) { \u0026#34;Post with title $title already exists\u0026#34; } val content = async { verifyContent(title, body) } val author = async { authorBy(authorId) } val post = Post(author.await(), content.await(), Date()) repository.save(post) } } /* Pretending to make a call to the Authorship Service that keeps track of authorized authors. */ private suspend fun authorBy(id: String): Author { ... } /* Some local computation that verifies the content of the post is appropriate. */ private suspend fun verifyContent(title: String, body: String): PostContent { ... } a coroutineScope is a suspending function used to create a new coroutine scope: it suspends the execution of the current coroutine, releasing the underlying thread for other usages; As we said previously, the failure of a child with an exception immediately cancels its parent and, consequently, all its other children: this means that, for handling the cancellation of nested coroutines, we don\u0026rsquo;t need to do anything special with coroutineScope no matter the order in which coroutines are awaited, if one of them fails with an exception it is propagated upwards, cancelling all other ones this is not the case for supervisorScope, a coroutine builder ensuring that child coroutines can fail independently without affecting the parent coroutine. have a look to this test This is an advantage over the Scala Gears, where operators like zip and altWithCancel are necessary! Takeaways # Scala Gears offers, despite the syntactical differences, very similar concepts to Kotlin Coroutines, with structured concurrency and cancellation mechanisms; Kotlin Coroutines handles the cancellation of nested coroutines more easily than Scala Gears, where special attention is required; As stated by M. Odersky the Async capability is better than suspend because let defines functions that work for synchronous as well as asynchronous function arguments without changing anything, while in Kotlin suspendable functions passed as an argument in higher-order functions must be tagged with suspend keyword. Previous: boundary \u0026amp; break Next: Channels as a communication primitive "},{"id":3,"href":"/direct-style-experiments/docs/04-channels/","title":"04 Channels","section":"Docs","content":" Channels as a communication primitive # Channels as a communication primitive Introduction Analyzer example Future monadic version Scala Gears version Kotlin Coroutines version Introducing Flows in Gears Showcasing Flows Takeaways Introduction # The fourth, yet not mentioned, abstraction of both Kotlin Coroutines and Scala Gears is the channel. Channels represent the primitive communication and coordination means to exchange Future (or coroutines in the case of Kotlin) results. They are, at least conceptually, very similar to a queue where it is possible to send (and receive) data \u0026ndash; basically, exploiting the producer-consumer pattern.\nclassDiagram class `SendableChannel[-T]` { \u003c\u003c trait \u003e\u003e +sendSource(x: T) Async.Source[Either[Closed, Unit]] +send(x: T)(using Async) Unit } class `ReadableChannel[+T]` { \u003c\u003c trait \u003e\u003e +readSource Async.Source[Either[Closed, T]] +read()(using Async) Either[Closed, T] } class `Channel[T]` { \u003c\u003c trait \u003e\u003e +asSendable: SendableChannel[T] +asReadable: ReadableChannel[T] +asCloseable: java.io.Closeable } namespace java io { class Closeable { \u003c\u003c interface \u003e\u003e +close() } } `SendableChannel[-T]` \u003c|-- `Channel[T]` Closeable \u003c|-- `Channel[T]` `ReadableChannel[+T]` \u003c|-- `Channel[T]` The channel is defined through three distinct interfaces: SendableChannel[-T], ReadableChannel[+T] and Channel[T], where the latter extends from both SendableChannel and ReadableChannel. Typically, a Channel is created and a SendableChannel and ReadableChannel instances are respectively provided to the producer and the consumer, restricting their access to it. The same, almost identical, design is present also in Kotlin Coroutines where SendChannel and ReceiveChannel take over, respectively, the Gears SendableChannel and ReadableChannel.\nChannel inherits from java.io.Closable, making them closable objects: once closed, they raise ChannelClosedException when attempting to write to them and immediately return a Left(Closed) when attempting to read from them, preventing the consumer from finishing reading all the values sent on the channel before its closing. This is not the case for Kotlin Coroutines where closing a channel indicates that no more values are coming, but doesn\u0026rsquo;t prevent consuming already sent values. Moreover, in Kotlin is possible to use a regular for loop to receive elements from a channel (blocking the coroutine): val channel = Channel\u0026lt;Int\u0026gt;() launch { for (x in 1..5) channel.send(x * x) channel.close() // we\u0026#39;re done sending } for (y in channel) println(y) // blocks until channel is closed println(\u0026#34;Done!\u0026#34;) Similar behavior can be achieved also in Gears extending the framework with the concept of Terminable channel. After all, closing a channel in coroutines is a matter of sending a special token to it, allowing stop the iteration as soon as this token is received. [The full implementation can be found in commons submodule, pimping package.]\n/** A token to be sent to a channel to signal that it has been terminated. */ case object Terminated type Terminated = Terminated.type /** A union type of [[T]] and [[Terminated]]. */ type Terminable[T] = T | Terminated /** Exception being raised by [[TerminableChannel.send()]] on terminated [[TerminableChannel]]. */ class ChannelTerminatedException extends Exception /** A [[Channel]] that can be terminated, signalling no more items will be sent, * still allowing to consumer to read pending values. * Trying to `send` values after its termination arise a [[ChannelTerminatedException]]. * When one consumer reads the [[Terminated]] token, the channel is closed. Any subsequent * read will return `Left(Channel.Closed`. */ trait TerminableChannel[T] extends Channel[Terminable[T]]: def terminate()(using Async): Unit object TerminableChannel: /** Creates a [[TerminableChannel]] backed to [[SyncChannel]]. */ def ofSync[T]: TerminableChannel[T] = TerminableChannelImpl(SyncChannel()) /** Creates a [[TerminableChannel]] backed to [[BufferedChannel]]. */ def ofBuffered[T]: TerminableChannel[T] = TerminableChannelImpl(BufferedChannel()) /** Creates a [[TerminableChannel]] backed to an [[UnboundedChannel]]. */ def ofUnbounded[T]: TerminableChannel[T] = TerminableChannelImpl(UnboundedChannel()) private class TerminableChannelImpl[T](c: Channel[Terminable[T]]) extends TerminableChannel[T]: opaque type Res[R] = Either[Channel.Closed, R] private var _terminated: Boolean = false override val readSource: Async.Source[Res[Terminable[T]]] = c.readSource.transformValuesWith { case Right(Terminated) =\u0026gt; c.close(); Left(Channel.Closed) case v @ _ =\u0026gt; v } override def sendSource(x: Terminable[T]): Async.Source[Res[Unit]] = synchronized: if _terminated then throw ChannelTerminatedException() else if x == Terminated then _terminated = true c.sendSource(x) override def close(): Unit = c.close() override def terminate()(using Async): Unit = try send(Terminated) // It happens only at the close of the channel due to the call (inside Gears library) of // a CellBuf.dequeue(channels.scala:239) which is empty! catch case _: NoSuchElementException =\u0026gt; () // e.printStackTrace() Now, thanks to this extension, also in Scala Gears is possible to write:\nval channel = TerminableChannel.ofUnbounded[Int] Future: (0 until 10).foreach(channel.send(_)) channel.terminate() // we\u0026#39;re done sending channel.foreach(println(_)) // blocks until channel is closed println(\u0026#34;Done!\u0026#34;) [Other tests can be found in TerminableChannelTest.]\nOn top of this new abstraction is possible to implement, for example, the foreach and toSeq methods, which can be useful to wait for all the items sent over the channel.\nobject TerminableChannelOps: extension [T: ClassTag](c: TerminableChannel[T]) /** Blocking consume channel items, executing the given function [[f]] for each element. */ @tailrec def foreach[U](f: T =\u0026gt; U)(using Async): Unit = c.read() match case Left(Channel.Closed) =\u0026gt; () case Right(value) =\u0026gt; value match case Terminated =\u0026gt; () case v: T =\u0026gt; f(v); foreach(f) /** @return a [[Seq]] containing channel items, after having them read. * This is a blocking operation! */ def toSeq(using Async): Seq[T] = var results = Seq[T]() c.foreach(t =\u0026gt; results = results :+ t) results Three types of channels exist:\nSynchronous Channels: links a read request with a send within a rendezvous send (read) suspend the process until a consumer read (send) the value; in Kotlin they are called Rendezvous Channels. Buffered Channels: a version of a channel with an internal buffer of fixed size send suspend the producer process if it is full; otherwise, it appends the value to the buffer, returning immediately; read suspend if the channel is empty, waiting for a new value. Unbounded Channels: a version of a channel with an unbounded buffer if the programs run out of memory you can get an out-of-memory exception! in Kotlin they are called Unlimited Channel. Kotlin offers also a fourth type: the Conflated Channel, where every new element sent to it overwrites the previously sent one, never blocking, so that the receiver gets always the latest element.\nConcerning channel behavior, it is important to note that:\nMultiple producers can send data to the channel, as well as multiple consumers can read them, but each element is handled only once, by one of them, i.e. consumers compete with each other for sent values; Once the element is handled, it is immediately removed from the channel; Channels are fair: send and read operations to channels are fair w.r.t. the order of their invocations from multiple threads (they are served in first-in first-out order). Analyzer example # To show channels in action an example has been prepared:\nIdea: we want to realize a little asynchronous library allowing clients to collect the common statistics about repositories (issues, stars, last release) and contributors of a given GitHub organization.\nThe final result is a GUI application that, given an organization name, starts the analysis of all its repositories, listing their information along with all their contributors as soon as they are computed. Moreover, the application allows the user to cancel the current computation at any point in time.\n[This example has been inspired by this tutorial.]\nTo start the application:\n./gradlew analyzer-\u0026lt;direct | direct-kt | monadic\u0026gt;:run In order to run the application you need to place inside the analyzer-commons directory a .env file containing your personal GitHub access token, like:\nGH_TOKEN=.... or having set an environment variable named GH_TOKEN.\nThe example is structured in two different packages: lib and client. The former contains the logic of the library, while the latter contains the application (client code). As usual, it has been implemented using monadic Futures, as well as using Scala Gears and Kotlin Coroutines.\nFuture monadic version # [The sources are available inside the analyzer-monadic submodule.]\nThe entry point of the library is the Analyzer interface which takes in input the organization name and a function through which is possible to react to results while they are computed.\nSince we want to achieve cancellation, the monadic version leverages Monix Task, which is returned by the analyze method wrapped in an EitherT monad transformer to allow handling errors functionally.\n/** A generic analyzer of organization/group/workspace repositories. */ trait Analyzer: /** @return a [[EitherT]] encapsulating a [[Task]] that performs the analysis of the * [[organizationName]]\u0026#39;s repositories, providing the results incrementally to the * [[updateResults]] function. */ def analyze(organizationName: String)( updateResult: RepositoryReport =\u0026gt; Unit, ): EitherT[Task, String, Seq[RepositoryReport]] To retrieve data from GitHub, a RepositoryService interface has been created, following the same pattern:\n/** A service exposing functions to retrieve data from a central hosting repository service. */ trait RepositoryService: /** @return a [[EitherT]] encapsulating a [[Task]] which get all the repositories * owned by [[organizationName]]. */ def repositoriesOf(organizationName: String): EitherT[Task, String, Seq[Repository]] /** @return a [[EitherT]] encapsulating a [[Task]] which get all the contributors * of [[repositoryName]] owned by [[organizationName]]. */ def contributorsOf(organizationName: String, repositoryName: String): EitherT[Task, String, Seq[Contribution]] /** @return a [[EitherT]] encapsulating a [[Task]] which get the last release * of [[repositoryName]] owned by [[organizationName]]. */ def lastReleaseOf(organizationName: String, repositoryName: String): EitherT[Task, String, Release] The implementation of the Analyzer is shown in the following code snippet and performs the following steps:\nfirst, the list of repositories is retrieved; if no error occurs, the analysis of each repository is performed concurrently, thanks to the parTraverse operator; the analysis of each repository consists of retrieving the contributors and the last release of the repository and then updating the result through the updateResult function. Since both the contributors and last release retrieval are independent of each other, they are performed concurrently, thanks to Task.parZip2. override def analyze(organizationName: String)( updateResult: RepositoryReport =\u0026gt; Unit, ): EitherT[Task, String, Seq[RepositoryReport]] = for repositories \u0026lt;- repositoryService.repositoriesOf(organizationName) reports \u0026lt;- repositories.parTraverse(r =\u0026gt; EitherT.right(r.performAnalysis(updateResult))) yield reports extension (r: Repository) private def performAnalysis(updateResult: RepositoryReport =\u0026gt; Unit): Task[RepositoryReport] = val contributorsTask = repositoryService.contributorsOf(r.organization, r.name).value val releaseTask = repositoryService.lastReleaseOf(r.organization, r.name).value for result \u0026lt;- Task.parZip2(contributorsTask, releaseTask) report = RepositoryReport(r.name, r.issues, r.stars, result._1.getOrElse(Seq.empty), result._2.toOption) _ \u0026lt;- Task(updateResult(report)) yield report Client-side, when a new session is requested, the Analyzer is used to start the computation, during which the single reports are aggregated and the UI is updated. Whenever desired, the current computation can be stopped by canceling the Monix CancelableFuture returned by the runToFuture method, through which the returned Task from the Analyzer is started.\nclass MonadicAppController extends AppController: import monix.execution.Scheduler.Implicits.global private val view = AnalyzerView.gui(this) private val analyzer = Analyzer(RepositoryService.ofGitHub) private var currentComputation: Option[CancelableFuture[Unit]] = None view.run() override def runSession(organizationName: String): Unit = var organizationReport: OrganizationReport = (Map(), Set()) val f = analyzer.analyze(organizationName) { report =\u0026gt; organizationReport = organizationReport.mergeWith(report) view.update(organizationReport) }.value.runToFuture.map { case Left(value) =\u0026gt; view.error(value); case Right(_) =\u0026gt; view.endComputation() } currentComputation = Some(f) override def stopSession(): Unit = currentComputation foreach (_.cancel()) Scala Gears version # [The sources are available inside the analyzer-direct submodule.]\nThe interfaces of the Direct Style with Gears differ from the monadic one by their return type, which is a simpler Either data type, and by the fact they are suspendable functions, hence they require an Async context to be executed. This is the first important difference: the analyze method, differently from the monadic version, doesn\u0026rsquo;t return immediately the control; instead, it suspends the execution of the client until the result is available (though offering the opportunity to react to each update). This obeys the principle of explicit asynchrony: if the client wants to perform this operation asynchronously, it has to opt in explicitly, using a Future.\n/** A generic analyzer of organization/group/workspace repositories. */ trait Analyzer: /** Performs a **suspending** analysis of the [[organizationName]]\u0026#39;s repositories, * providing the results incrementally to the [[updateResults]] function. * @return [[Right]] with the overall results of the analysis or [[Left]] with * an error message in case of failure. */ def analyze(organizationName: String)( updateResults: RepositoryReport =\u0026gt; Unit, )(using Async, AsyncOperations): Either[String, Seq[RepositoryReport]] /** A service exposing functions to retrieve data from a central hosting repository service. */ trait RepositoryService: /** @return [[Right]] with the [[Seq]]uence of [[Repository]] owned by the given * [[organizationName]] or a [[Left]] with a explanatory message in case of errors. */ def repositoriesOf(organizationName: String)(using Async): Either[String, Seq[Repository]] /** @return [[Right]] with the [[Seq]]uence of [[Contribution]] for the given * [[repositoryName]] owned by the given [[organizationName]] or a [[Left]] with an * explanatory message in case of errors. */ def contributorsOf( organizationName: String, repositoryName: String )(using Async): Either[String, Seq[Contribution]] /** @return a [[Right]] with the last [[Release]] of the given [[repositoryName]] owned * by [[organizationName]] if it exists, or a [[Left]] with a explanatory message * in case of errors. */ def lastReleaseOf( organizationName: String, repositoryName: String )(using Async): Either[String, Release] The implementation of the Analyzer leverages Channels to perform the concurrent analysis of the repositories:\noverride def analyze(organizationName: String)( updateResults: RepositoryReport =\u0026gt; Unit, )(using Async, AsyncOperations): Either[String, Seq[RepositoryReport]] = either: val reposInfo = repositoryService.repositoriesOf(organizationName).? // 1 .map(_.performAnalysis.run) // 2 val collector = Collector[RepositoryReport](reposInfo.toList*) // 3 reposInfo.foreach(_ =\u0026gt; updateResults(collector.results.read().asTry.?.awaitResult.?) // 4 ) reposInfo.awaitAll // 5 extension (r: Repository) protected def performAnalysis(using Async): Task[RepositoryReport] = Task: val contributions = Future: // concurrent! repositoryService.contributorsOf(r.organization, r.name) val release = repositoryService.lastReleaseOf(r.organization, r.name) RepositoryReport(r.name, r.issues, r.stars, contributions.await.getOrElse(Seq()), release.toOption) first, we get all the repositories of the requested organization for each of them, the contributors and the last release are retrieved concurrently, starting a Future Future results are gathered inside a Collector allowing to collect a list of futures into a channel of futures, arriving as they finish. the retrieval of the contributors and the last release are performed in parallel read results from the channel as they come, calling the updateResult reaction function. Overall results are returned to the client. Although it works, the proposed solution suffers from a performance issue when the organization we want to analyze has a large number of repositories. Indeed, the GitHub API, like many ReSTful APIs, implements pagination: if the response includes many results, they are paginated, returning a subset of them; it is the responsibility of the client to request more data (pages). Until now, the RepositoryService has been implemented to return the whole results in one shot, leading to suspension until all pages are retrieved. It would be desirable, instead, to start performing the analysis as soon as one page is obtained from the API.\nTo do so, the interface of the RepositoryService has been extended with new methods, incremental***, returning a TerminableChannel of results:\ntrait RepositoryService: /** @return a [[Terminable]] [[ReadableChannel]] with the [[Repository]] owned by the given * [[organizationName]], wrapped inside a [[Either]] for errors management. */ def incrementalRepositoriesOf( organizationName: String, )(using Async): TerminableChannel[Either[String, Repository]] /** @return a [[Terminable]] [[ReadableChannel]] with the [[Contribution]] made by users to the * given [[repositoryName]] owned by [[organizationName]], wrapped inside a [[Either]]. */ def incrementalContributorsOf( organizationName: String, repositoryName: String, )(using Async): TerminableChannel[Either[String, Contribution]] // ... Then, the implementation of the analyze method becomes:\noverride def analyze(organizationName: String)( updateResults: RepositoryReport =\u0026gt; Unit, )(using Async, AsyncOperations): Either[String, Seq[RepositoryReport]] = either: val reposInfo = repositoryService.incrementalRepositoriesOf(organizationName) // 1 var futures = Seq[Future[RepositoryReport]]() reposInfo.foreach { repository =\u0026gt; // 2 futures = futures :+ Future: // 3 val report = repository.?.performAnalysis.run.awaitResult.? updateResults(report) report } futures.awaitAllOrCancel we get the channel of repositories from the repository service; the foreach method of TerminableChannel is used to iterate over all the repositories sent over the channel as soon as they are retrieved by the service. This is a blocking operation, i.e. it suspends until all the repositories are retrieved; we start the analysis in a separate Future (i.e. thread): this allows you to start the analysis as soon as a repository is fetched by the channel, preventing starting the analysis of the next repository only when the previous one is finished; once all the repositories are retrieved, i.e. the foreach terminates, we wait for the completion of all the started Futures. Indeed, when the foreach terminates, we have the guarantee that all started futures have been started, but not yet completed! Kotlin Coroutines version # [The sources are available inside the analyzer-direct-kt submodule.]\nThe analyzer interface reflects the Scala Gears one: a Result is used in place of Either, and the suspendable function udateResults is marked with the suspend keyword in place of the using Async context.\ninterface Analyzer { suspend fun analyze( organizationName: String, updateResults: suspend (RepositoryReport) -\u0026gt; Unit = { _ -\u0026gt; }, ): Result\u0026lt;Set\u0026lt;RepositoryReport\u0026gt;\u0026gt; } Its channel-based implementation, despite syntactic differences, is also very similar to that of Scala Gears, at least conceptually:\nwe get all the repositories; for each of them, an analysis is started to retrieve the contributors and the last release; each analysis is started in a separate coroutine whose results are sent to a channel; as usual, the contributors and the last release are retrieved concurrently, using the async coroutine builder; results are aggregated as they come from the channel. override suspend fun analyze( organizationName: String, updateResults: suspend (RepositoryReport) -\u0026gt; Unit, ): Result\u0026lt;Set\u0026lt;RepositoryReport\u0026gt;\u0026gt; = coroutineScope { runCatching { val repositories = provider.repositoriesOf(organizationName).getOrThrow() // 1 val resultsChannel = analyzeAll(organizationName, repositories) // 2 collectResults(resultsChannel, repositories.size, updateResults) // 3 } } private fun CoroutineScope.analyzeAll(organizationName: String, repositories: List\u0026lt;Repository\u0026gt;) = Channel\u0026lt;RepositoryReport\u0026gt;().also { repositories.map { r -\u0026gt; launch { // a new coroutine for each repository is started val contributors = async { provider.contributorsOf(organizationName, r.name).getOrThrow() } val release = provider.lastReleaseOf(organizationName, r.name).getOrThrow() it.send(RepositoryReport(r.name, r.issues, r.stars, contributors.await(), release)) } } } private suspend fun collectResults( resultsChannel: Channel\u0026lt;RepositoryReport\u0026gt;, expectedResults: Int, updateResults: suspend (RepositoryReport) -\u0026gt; Unit, ) = mutableSetOf\u0026lt;RepositoryReport\u0026gt;().apply { repeat(expectedResults) { val report = resultsChannel.receive() add(report) updateResults(report) } resultsChannel.close() } Where, instead, Kotlin Coroutines shine is the implementation of the RepositoryService for supporting incremental retrieval of repositories and contributors.\nIndeed, Kotlin has a built-in support for cold streams, called Flow. They are very similar (actually they have been inspired to) cold observable in reactive programming, and they are the perfect fit for functions that need to return a stream of asynchronously computed values.\nThe RepositoryService has been here extended with new methods, flowing***, returning a Flow of results:\nclass GitHubRepositoryProvider { fun flowingRepositoriesOf(organizationName: String): Flow\u0026lt;List\u0026lt;Repository\u0026gt;\u0026gt; fun flowingContributorsOf(organizationName: String, repositoryName: String): Flow\u0026lt;List\u0026lt;Contribution\u0026gt;\u0026gt; } As already mentioned, the Flow is a cold stream, meaning that it is not started until it is collected. Once the collect method is called a new stream is created and data starts to \u0026ldquo;flow\u0026rdquo;.\nThey offer several useful operators for transforming and combining them functionally (not a complete list):\nIntermediate flow operators:\nfilter/filterNot to filter out unwanted values; map to transform the values; transform to implement more complex transformations (possibly involving suspending operations); take and its variant (e.g. takeWhile) to limit the number of values emitted; onEach to perform side-effects for each value emitted. Terminal flow operators:\nconversions to various collection types, like toList, toSet; first, last, single to retrieve the first, last or single value emitted; reduce to perform some kind of operation over all items, reducing them to a single one; fold to perform some kind of operation over all items, starting from an initial value, accumulating a result. Flows combining operators:\nmerge to combine multiple flows into a single one, emitting values from all of them; zip combines the corresponding values of two flows; flatMapConcat / flatMapMerge to transform each value into a flow and then concatenate/merge them. Moreover, like in Rx:\nit is possible to control the context in which the flow is executed using the flowOn operator, which changes the context for all the steps above it (so it is typically used as the last step in a function); some backpressure strategies are supported, which are used to handle the situation when the producer is emitting values faster than the consumer can process them. In Kotlin Coroutines, the backpressure is managed by the buffer (and its variant, like conflated, sample, debounce) operator, which allows to buffer a certain number of values before the consumer starts to process them. override suspend fun analyze( organizationName: String, updateResults: suspend (RepositoryReport) -\u0026gt; Unit, ): Result\u0026lt;Set\u0026lt;RepositoryReport\u0026gt;\u0026gt; = coroutineScope { runCatching { val reports = provider.flowingRepositoriesOf(organizationName) .flatMapConcat { analyzeAll(it) } .flowOn(Dispatchers.Default) var allReports = emptySet\u0026lt;RepositoryReport\u0026gt;() // until here just \u0026#34;configuration\u0026#34; reports.collect { updateResults(it) allReports = allReports + it } allReports } } Concerning flows, an important thing to note is that they are just asynchronous generators that run some suspending code when you collect them. Thus, per se, they don\u0026rsquo;t introduce new coroutines or concurrency mechanism. To achieve concurrency, for example emitting values concurrently by multiple coroutines, is necessary to use a channelFlow, a pre-cooked way to inject a coroutineContext and a Channel through which is possible to pass the values to be emitted from a background coroutines back to the main control flow:\nfun analyzeAll(repositories: List\u0026lt;Repository\u0026gt;): Flow\u0026lt;RepositoryReport\u0026gt; = channelFlow { repositories.forEach { repository -\u0026gt; launch { val release = async { provider.lastReleaseOf(repository.organization, repository.name).getOrThrow() } provider.flowingContributorsOf(repository.organization, repository.name).toList().forEach { // emit this value send(RepositoryReport(repository.name, repository.issues, repository.stars, it, release.await())) } } } } Introducing Flows in Gears # A similar abstraction of Kotlin Flows can be implemented in Scala Gears leveraging Tasks and TerminableChannels. The following section describes the attempt made to implement it and what has been achieved. When building the Flow, the client provides a block of code through which emits values, which is wrapped inside a Task that is started only when the collect method is called; The values are sent (emitted) on a TerminableChannel which is created when the collect method is called; the behavior of the emit method is defined inside the apply method of Flow and injected inside caller code via the context parameter (it: FlowCollector[T]) ?=\u0026gt;. Once the task has finished, the channel is terminated. [Source code can be found in commons submodule, pimpimg package.]\n/** An asynchronous cold data stream that emits values, inspired to Kotlin Flows. */ trait Flow[+T]: /** Start the flowing of data which can be collected reacting through the given [[collector]] function. */ def collect(collector: Try[T] =\u0026gt; Unit)(using Async, AsyncOperations): Unit /** An interface modeling an entity capable of [[emit]]ting [[Flow]]s values. */ trait FlowCollector[-T]: /** Emits a value to the flow. */ def emit(value: T)(using Async): Unit object Flow: /** Creates a new asynchronous cold [[Flow]] from the given [[body]]. * Since it is cold, it starts emitting values only when the [[Flow.collect]] method is called. * To emit a value use the [[FlowCollector]] given instance. */ def apply[T](body: (it: FlowCollector[T]) ?=\u0026gt; Unit): Flow[T] = val flow = FlowImpl[T]() flow.task = Task: val channel = flow.channel flow.sync.release() val collector: FlowCollector[T] = new FlowCollector[T]: override def emit(value: T)(using Async): Unit = channel.send(Success(value)) try body(using collector) catch case e: Exception =\u0026gt; channel.send(Failure(e)) flow private class FlowImpl[T] extends Flow[T]: private[Flow] var task: Task[Unit] = uninitialized private[Flow] var channel: TerminableChannel[Try[T]] = uninitialized private[Flow] val sync = Semaphore(0) override def collect(collector: Try[T] =\u0026gt; Unit)(using Async, AsyncOperations): Unit = val myChannel = TerminableChannel.ofUnbounded[Try[T]] synchronized: channel = myChannel task.run.onComplete(() =\u0026gt; myChannel.terminate()) // Ensure to leave the synchronized block after the task has been initialized // with the correct channel instance. sync.acquire() myChannel.foreach(t =\u0026gt; collector(t)) map and flatMap combinators have been implemented on top of Flow:\nobject FlowOps: extension [T](flow: Flow[T]) /** @return a new [[Flow]] whose values has been transformed according to [[f]]. */ def map[R](f: T =\u0026gt; R): Flow[R] = new Flow[R]: override def collect(collector: Try[R] =\u0026gt; Unit)(using Async, AsyncOperations): Unit = catchFailure(collector): flow.collect(item =\u0026gt; collector(Success(f(item.get)))) /** @return a new [[Flow]] whose values are created by flattening the flows generated * by the given function [[f]] applied to each emitted value of this. */ def flatMap[R](f: T =\u0026gt; Flow[R]): Flow[R] = new Flow[R]: override def collect(collector: Try[R] =\u0026gt; Unit)(using Async, AsyncOperations): Unit = catchFailure(collector): flow.collect(item =\u0026gt; f(item.get).collect(x =\u0026gt; collector(Success(x.get)))) Showcasing Flows # Library use case:\ntype Name = String type WriterId = Int type Writer = (Name, WriterId) type Book = String object LibraryService: private val users: Set[Writer] = Set((\u0026#34;Alice\u0026#34;, 987), (\u0026#34;Bob\u0026#34;, 123), (\u0026#34;Charlie\u0026#34;, 342)) private val books: Map[WriterId, Set[Book]] = Map( 987 -\u0026gt; Set(\u0026#34;Alice\u0026#39;s Adventures in Wonderland\u0026#34;, \u0026#34;Through the Looking-Glass\u0026#34;), 123 -\u0026gt; Set(\u0026#34;The Shining\u0026#34;), 342 -\u0026gt; Set(\u0026#34;The Raven\u0026#34;, \u0026#34;The Tell-Tale Heart\u0026#34;), ) def allWriters(using Async): Flow[Writer] = Flow: users.foreach { u =\u0026gt; sleep(2_000) it.emit(u) } def booksByWriter(writer: WriterId)(using Async): Flow[Book] = Flow: books(writer).foreach(it.emit) Flows are cold\n@main def useSimple(): Unit = Async.blocking: val service = LibraryService val writers = service.allWriters log(s\u0026#34;Not collecting yet!\u0026#34;) sleep(1_000) // something meaningful log(\u0026#34;Starting collecting users...\u0026#34;) writers.collect(u =\u0026gt; log(s\u0026#34;User: $u\u0026#34;)) println(\u0026#34;Done\u0026#34;) What we get is something like:\n[1709559932492] Not collecting yet! [1709559933500] Starting collecting users... [1709559935532] User: Success((Alice,987)) [1709559937536] User: Success((Bob,123)) [1709559939541] User: Success((Charlie,342)) Done If something goes wrong during the task execution, a Failure is emitted and the task terminates (no more values are emitted). For example:\ndef failingWriters(using Async): Flow[Writer] = Flow: throw IllegalStateException(\u0026#34;Library is closed\u0026#34;) it.emit(users.head) @main def useFailingFlow(): Unit = Async.blocking: val service = LibraryService val writers = service.failingWriters writers.collect(println) Results in:\nFailure(java.lang.IllegalStateException: The library is closed) Flows can be transformed\nmapping Writers to their identifier:\n@main def useWithMapping(): Unit = Async.blocking: val service = LibraryService val writersId = service.allWriters.map(_._2) writersId.collect(a =\u0026gt; println(s\u0026#34;Id: $a\u0026#34;)) Result:\nId: Success(987) Id: Success(123) Id: Success(342) flatMapping to get all the books:\n@main def useWithFlatMap(): Unit = Async.blocking: val service = LibraryService val allBooks = service.allWriters.flatMap(w =\u0026gt; service.booksByWriter(w._2) ) allBooks.collect(println) Result:\nSuccess(Alice\u0026#39;s Adventures in Wonderland) Success(Through the Looking-Glass) Success(The Shining) Success(The Raven) Success(The Tell-Tale Heart) 👉🏻 More tests on Flows can be found in commons, pimping pakcage.\nTakeaways # Channels are the basic communication and synchronization primitive for exchanging data between Futures/Coroutines. Scala Gears support for Terminable channels or a review of the closing mechanism should be considered. The Flow abstraction in Kotlin Coroutines is a powerful tool for handling cold streams of data, and it is a perfect fit for functions that need to return a stream of asynchronously computed values upon request. A similar abstraction can be implemented in Scala Gears leveraging Tasks and TerminableChannels, enabling improved support for an asynchronous flow of data also in Gears, which is currently lacking. Previous: Basic asynchronous constructs Next: Reactivity in direct style "},{"id":4,"href":"/direct-style-experiments/docs/05-rears/","title":"05 Rears","section":"Docs","content":" Reactivity in direct style # So far, we\u0026rsquo;ve explored the basics of asynchronous abstraction mechanisms provided by the direct style of the Scala Gears and Kotlin Coroutines frameworks. The goal of this last example is to investigate, using a simple example, whether these two frameworks offer sufficient idiomatic abstractions to deal with event-based reactive systems.\nSmart Hub example # Idea: in an IoT context, a multitude of sensors of different types, each replicated to ensure accuracy, transmit their measurements to a central hub, which in turn needs to react, in real-time, forwarding to the appropriate controller the data, possibly performing some kind of transformation. Scala Gears version # Before delving into the example, two abstractions of Gears, yet not covered, are introduced:\nTasks provide a way, not only to run asynchronous computation, essentially wrapping a () =\u0026gt; Future[T], but also to schedule it, possibly repeating it. Different scheduling strategies are available: Every, ExponentialBackoff, FibonacciBackoff, RepeatUntilFailure, RepeatUntilSuccess. This allows for implementing easily proactive computations classDiagram class `Task[+T]` { +apply(body: (Async, AsyncOperations) ?=\u003e T) Task[T]$ +run(using Async, AsyncOperations) Future[+T] +schedule(s: TaskSchedule) Task[T] } `Task[+T]` o--\u003e TaskSchedule class TaskSchedule { \u003c\u003c enum \u003e\u003e + Every(millis: Long, maxRepetitions: Long = 0) + ExponentialBackoff(millis: Long, exponentialBase: Int = 2, maxRepetitions: Long = 0) + FibonacciBackoff(millis: Long, maxRepetitions: Long = 0) + RepeatUntilFailure(millis: Long = 0, maxRepetitions: Long = 0) + RepeatUntilSuccess(millis: Long = 0, maxRepetitions: Long = 0) } Warning: when Tasks are scheduled with RepeatUntil*:\nif the body of a Task does not perform any suspending operations the Async.blocking blocks the current thread until the task is completed (either successfully or not); if the body of a Task does perform suspending operations then the Async.blocking does not wait for the task to complete and its context is left as soon as reaches its end. If we want to wait for the task completion, it\u0026rsquo;s the client\u0026rsquo;s responsibility to explicitly Async.await (or awaitResult) Cons: depending on the content of the block, the behavior is different! This is error-prone! Warning: with high-order functions if we deal with repeated Tasks, in some cases an Async ?=\u0026gt; label is required to not suspend the whole block, even if a suspending operation is performed: the code below behaves differently if the Async ?=\u0026gt; label is present or not. Note: this may be an unintended effect of the library, yet to be investigated (sometimes on Ubuntu it doesn\u0026rsquo;t work, suggesting to be a \u0026ldquo;bug\u0026rdquo;, see here vs. here) In this case despite we suspend to wait for the timer tick, the Async.blocking blocks until the Task is completed.\nAsync.blocking: val timer = Timer(2.seconds) Future(timer.run()) produce { _ =\u0026gt; timer.src.awaitResult // SUSPENDING OPERATION! // ... } def produce[T]( action: SendableChannel[T] =\u0026gt; Try[Unit] )(using Async): ReadableChannel[T] = val channel = UnboundedChannel[T]() Task { action(channel.asSendable) }.schedule(RepeatUntilFailure()).run channel.asReadable With the Async ?=\u0026gt; label, the Async.blocking does not wait for the Task to complete!\nAsync.blocking: val timer = Timer(2.seconds) Future(timer.run()) produceWithLabel { _ =\u0026gt; timer.src.awaitResult // SUSPENDING OPERATION! // .... } def produceWithLabel[T]( action: Async ?=\u0026gt; SendableChannel[T] =\u0026gt; Try[Unit] )(using Async): ReadableChannel[T] = val channel = UnboundedChannel[T]() Task { action(channel.asSendable) }.schedule(RepeatUntilFailure()).run channel.asReadable [See the tests for more details.]\nTo avoid the work-stealing behavior of channel consumers, a ChannelMultiplexer can be used. It is essentially a container of Readable and Sendable channels, which can be added and removed at runtime. Internally, it is implemented with a thread that continuously races the set of publishers and once it reads a value, it forwards it to each subscriber channel. Order is guaranteed only per producer; Typically, the consumer creates a channel and adds it to the multiplexer, then starts reading from it, possibly using a scheduled task. if the consumer attaches to the channel after the producer has started, the values sent during this interval are lost, like hot observables in Rx. classDiagram namespace javaio { class Closeable { \u003c\u003c interface \u003e\u003e +close() } } class `ChannelMultiplexer[T]` { \u003c\u003c trait \u003e\u003e +run()(using Async) +addPublisher(c: ReadableChannel[T]) +removePublisher(c: ReadableChannel[T]) +addSubscriber(c: SendableChannel[Try[T]]) +removeSubscriber(c: SendableChannel[Try[T]]) } Closeable \u003c|-- `ChannelMultiplexer[T]` In the proposed strawman Scala Gears library, there are no other kinds of abstractions, nor a way to manipulate channels with functions inspired by Rx.\nThe attempt, described in the following, has been to extend this framework adding first-class support for Producer and Consumer\u0026rsquo;s concepts and implementing some of the most common Rx operators, completely leaving out performance concerns.\n[Sources can be found in the rears submodule.].\nA Producer is a runnable entity, programmed with a Task, producing items on a channel. It exposes the publishingChannel method, which returns a ReadableChannel through which interested consumers can read produced items. A Consumer is a runnable entity devoted to consuming data from a channel, exposed by the listeningChannel method which returns a SendableChannel to send items to. It can be made stateful by mixing it with the State trait, allowing it to keep track of its state, which is updated every time with the result of the reaction (i.e. its return type). Warning Like in an event-loop, the reaction logic should not perform long-lasting blocking operation, otherwise, the whole system will not react to new events: the Async capability is though needed if you want to give the client the ability to invoke Futures within this block; otherwise, another option (alternative to the following) would be to encapsulate the reaction behavior within a Task and run it at every received data. However, race conditions could take place in this last case. /** A producer, i.e. a runnable entity producing items on a channel. */ trait Producer[E]: /** The [[Channel]] where specific [[Producer]]s send items to. */ protected val channel: Channel[E] = UnboundedChannel() /** @return the publisher\u0026#39;s behavior encoded as a runnable [[Task]]. */ def asRunnable: Task[Unit] /** @return the [[ReadableChannel]] where produced items are placed. */ def publishingChannel: ReadableChannel[E] = channel.asReadable /** A consumer, i.e. a runnable entity devoted to consume data from a channel. */ trait Consumer[E, S]: /** The [[SendableChannel]] to send items to, where the consumer listen for new items. */ val listeningChannel: SendableChannel[Try[E]] = UnboundedChannel() /** @return a runnable [[Task]]. */ def asRunnable: Task[Unit] = Task { listeningChannel.asInstanceOf[Channel[Try[E]]].read().foreach(react) }.schedule(RepeatUntilFailure()) /** The suspendable reaction triggered upon a new read of an item succeeds. */ protected def react(e: Try[E])(using Async): S /** A mixin to make consumer stateful. Its state is updated with the result * of the [[react]]ion. Initially its state is set to [[initialValue]]. */ trait State[E, S](initialValue: S): consumer: Consumer[E, S] =\u0026gt; private var _state: S = initialValue /** @return the current state of the consumer. */ def state: S = synchronized(_state) override def asRunnable: Task[Unit] = Task { listeningChannel.asInstanceOf[Channel[Try[E]]].read().foreach { e =\u0026gt; synchronized { _state = react(e) } } }.schedule(RepeatUntilFailure()) The Controller object exposes methods wiring Producer and Consumers altogether, possibly performing some kind of transformation on the publisherChannel. the oneToOne method just wires one single consumer to the publisherChannel given in input, possibly having it transformed with the provided transformation. the oneToMany allows many consumers to be wired to the publisherChannel, possibly having it transformed. to accomplish this, a ChannelMultiplexer is used, which is in charge of forwarding the items read from the transformed publisherChannel to all consumers\u0026rsquo; channels. /** Simply, a function that, given in input a [[ReadableChannel]], performs some * kind of transformation, returning, as a result, another [[ReadableChannel]]. */ type PipelineTransformation[T, R] = ReadableChannel[T] =\u0026gt; ReadableChannel[R] object Controller: /** Creates a runnable [[Task]] forwarding the items read from the [[publisherChannel]] * to the given [[consumer]], after having it transformed with the given [[transformation]]. */ def oneToOne[T, R]( publisherChannel: ReadableChannel[T], consumer: Consumer[R, ?], transformation: PipelineTransformation[T, R] = identity, ): Task[Unit] = val transformedChannel = transformation(publisherChannel) Task { consumer.listeningChannel.send(transformedChannel.read().tryable) }.schedule(RepeatUntilFailure()) /** Creates a runnable [[Task]] forwarding the items read from the [[publisherChannel]] to * all consumers\u0026#39; channels, after having it transformed with the given [[transformation]]. */ def oneToMany[T, R]( publisherChannel: ReadableChannel[T], consumers: Set[Consumer[R, ?]], transformation: PipelineTransformation[T, R] = identity, ): Task[Unit] = Task: val multiplexer = ChannelMultiplexer[R]() consumers.foreach(c =\u0026gt; multiplexer.addSubscriber(c.listeningChannel)) multiplexer.addPublisher(transformation(publisherChannel)) // blocking call: the virtual thread on top of which this task is // executed needs to block to continue publishing publisher\u0026#39;s events // towards the consumer by means of the multiplexer. multiplexer.run() The following PipelineTransformations have been implemented (inspired by Rx). Tests in rears submodule provide the necessary examples to understand their behavior.\nFilter # /** @return a new [[ReadableChannel]] whose elements passes the given predicate [[p]]. */ def filter(p: T =\u0026gt; Boolean): ReadableChannel[T] Example:\n----1---2-------3----4---5--6----7---8---9---10---\u0026gt; | | | | | | | | | | ----V---V-------V----V---V--V----V---V---V---V----- filter(_ % 2 == 0) --------|--------------|------|-------|---------|-- V V V V V --------2--------------4------6-------8--------10-\u0026gt; Map # /** @return a new [[ReadableChannel]] whose values are transformed accordingly to the given function [[f]]. */ def map[R](f: T =\u0026gt; R): ReadableChannel[R] Example:\n----1---2-------3----4---5------6--------7--------\u0026gt; | | | | | | | ----V---V-------V----V---V------V--------V--------- map(x =\u0026gt; x * x) ----|---|-------|----|---|------|--------|--------- V V V V V V V ----1---4-------9----16--25-----36-------49-------\u0026gt; Debounce # /** @return a new [[ReadableChannel]] whose elements are emitted only after * the given [[timespan]] has elapsed since the last emission. */ def debounce(timespan: Duration): ReadableChannel[T] Example:\n----1---2-------3----4---5--6-----7---8---9---10--\u0026gt; | | | | | | | | | | V V V V V V V V V V T----------T----------T----------T----------T------ debounce(1 second) --------------------------------------------------- | | | | | V V V V V -------1---------3---------5------7------------10-\u0026gt; GroupBy # /** Groups the items emitted by a [[ReadableChannel]] according to the given [[keySelector]]. * @return key-value pairs, where the keys are the set of results obtained from applying the * [[keySelector]] coupled to a new [[ReadableChannel]] where only items belonging to * that grouping are emitted. */ def groupBy[K](keySelector: T =\u0026gt; K): ReadableChannel[(K, ReadableChannel[T])] Example:\n----1---2-3--4---5--6---\u0026gt; | | | | | | V V V V V V ------------------------- groupBy(_ % 2) ------------------------- \\ \\ ----false--true------------\u0026gt; 1 2 \\ \\ \\ 4 3 \\ \\ \\ 5 6 Buffer # /** @return a new [[ReadableChannel]] whose elements are buffered in a [[List]] of size [[n]]. * If [[timespan]] duration is elapsed since last read the list is emitted * with collected elements until that moment (default: 5 seconds). */ def buffer(n: Int, timespan: Duration = 5 seconds): ReadableChannel[List[T]] Example:\n----1---2-3----4---5--6----7---8--------\u0026gt; | | | | | | | | V V V V V V V V |---------|-----------|------------T----- buffer(n = 3, timespan = 5 seconds) |---------|-----------|------------|----- V V V ------[1, 2, 3]---[4, 5, 6]------[7, 8]-\u0026gt; BufferWithin # /** @return a new [[ReadableChannel]] whose elements are buffered in a [[List]] of items * if emitted within [[timespan]] duration after the first one (default: 5 seconds). */ def bufferWithin(timespan: Duration = 5 seconds): ReadableChannel[List[T]] Example:\n----1---2-3-4---5--6--7----------8-----------\u0026gt; | | | | | | | | V V V V V V V V ----|--------T--|--------T-------|--------T--- buffer(timespan = 5 seconds) -------------|-----------|----------------|--- V V V -------[1, 2, 3, 4]--[5, 6, 7]-----------[8]-\u0026gt; Going back to the example here is presented a schema summarizing the flows of data and the transformations to apply to them. This is just a simple example used to test the proposed abstractions.\n[Sources are available in smart-hub-direct submodule].\nFor simplicity, two types of sensors are considered: TemperatureSensor and LuminositySensor;\nsensors send data to the smart hub SensorSource (e.g., in a real case scenario, via MQTT)\nSensorSource is a Producer[SensorEvent], publishing received data on its publishingChannel:\ntrait SensorSource extends Producer[SensorEvent] sealed trait SensorEvent(val name: String) case class TemperatureEntry(sensorName: String, temperature: Temperature) extends SensorEvent(sensorName) case class LuminosityEntry(sensorName: String, luminosity: Temperature) extends SensorEvent(sensorName) three main controllers:\nSensorHealthChecker is a stateful consumer of generic SensorEvents that checks the health of the sensors, sending alerts in case of malfunctioning. Here the state is necessary to determine the health of the sensors, based on the last detection.\n/** A [[state]]ful consumer of [[SensorEvent]] detecting possible * malfunctioning and keeping track of last known active sensing units. */ trait SensorHealthChecker extends Consumer[Seq[E], Seq[E]] with State[Seq[E], Seq[E]] The Thermostat is a stateful consumer of temperature entries, taking care of controlling the heating system. The fact the thermostat keeps track of the last average detection could be useful to a ReSTful API, for example.\n/** A [[state]]ful consumer of [[TemperatureEntry]]s in charge of controlling * the heater and keeping track of the last detected average temperature. */ trait Thermostat extends Consumer[Seq[TemperatureEntry], Option[Temperature]] with State[Seq[TemperatureEntry], Option[Temperature]]: val scheduler: T LightingSystem is a basic consumer (non-stateful) of luminosity entries, taking care of controlling the lighting system.\n/** A consumer of [[LuminosityEntry]], in charge of controlling the lighting system. */ trait LightingSystem extends Consumer[Seq[LuminosityEntry], Unit] Each of these controllers reacts to the data received based on their logic and their actual state to accomplish a specific task. For example:\nthe sensor checker sends alerts whether, compared with the previous detection, it did not receive data from some sensors:\noverride protected def react(e: Try[Seq[E]])(using Async): Seq[E] = e match case Success(current) =\u0026gt; val noMoreActive = state.map(_.name).toSet -- current.map(_.name).toSet if noMoreActive.nonEmpty then sendAlert(s\u0026#34;[${currentTime}] Detected ${noMoreActive.mkString(\u0026#34;, \u0026#34;)} no more active!\u0026#34;) current case Failure(es) =\u0026gt; sendAlert(es.getMessage); Seq() the thermostat computes the average temperature and, based on a scheduler, decides whether to turn on or off the heating system:\noverride protected def react(e: Try[Seq[TemperatureEntry]])(using Async): Option[Temperature] = for averageTemperature \u0026lt;- e.map { entries =\u0026gt; entries.map(_.temperature).sum / entries.size }.toOption _ = averageTemperature.evaluate() // here logic to decide whether turn on or off heating system yield averageTemperature The HubManager takes care of grouping sensor data by their type and forwarding them to the appropriate manager, either ThermostatManager or LightingManager:\nval channelBySensor = sensorsSource.publishingChannel.groupBy(_.getClass) Task { channelBySensor.read() match case Right((clazz, c)) if clazz == classOf[TemperatureEntry] =\u0026gt; thermostatManager.run(c.asInstanceOf[ReadableChannel[TemperatureEntry]]) case Right((clazz, c)) if clazz == classOf[LuminosityEntry] =\u0026gt; lightingManager.run(c.asInstanceOf[ReadableChannel[LuminosityEntry]]) case _ =\u0026gt; () }.schedule(RepeatUntilFailure()).run sensorsSource.asRunnable.run.await Both ThermostatManager and LightingManager are in charge of creating the appropriate Controller instance, based on the number of Consumers and pipeline transformation we need to implement:\n// ThermostatManager def run(source: ReadableChannel[TemperatureEntry])(using Async, AsyncOperations): Unit = thermostat.asRunnable.run sensorHealthChecker.asRunnable.run Controller.oneToMany( publisherChannel = source, consumers = Set(thermostat, sensorHealthChecker), transformation = r =\u0026gt; r.bufferWithin(samplingWindow), ).run To produce a testable version of this example, a simulated source of sensor data has been created, backed to a GUI, through which the user can simulate the behavior of the sensors. The example is runnable via:\n./gradlew smart-hub-\u0026lt;direct | direct-kt\u0026gt;:run Three panels should pop up, one for each sensor type, and a dashboard showing the state of the system. Entering some value in the panels and pressing the \u0026ldquo;Send\u0026rdquo; button, after 10 seconds (the configured sampling window), the system should react to the data received, updating the dashboard with the new state.\nKotlin Coroutines version # Kotlin Coroutines offers two other abstractions to deal with asynchronous data streams, belonging to the flow \u0026ldquo;family\u0026rdquo;, which are: SharedFlow and StateFlow. Despite their names including flow, which we\u0026rsquo;ve seen are cold streams, they are actually hot (the terminology is a bit misleading\u0026hellip;):\nSharedFlow is a hot flow that allows for multiple collectors to subscribe to it, enabling the broadcasting of values to multiple consumers or having multiple consumers be \u0026ldquo;attached\u0026rdquo; to the same stream of data. they can be configured to buffer a certain number of previously emitted values for new collectors so that they can catch up with the latest values \u0026ndash; the so-called, replay cache; StateFlow is an extension of the SharedFlow: it is a hot flow that maintains a single value representing a state, holding one value at a time. It operates as a conflated flow, meaning that when a new value is emitted, it replaces the previous value and is immediately sent to new collectors this type of flow is beneficial for maintaining a single source of truth for a state and automatically updating all collectors with the latest state (for example in ViewModels in Android applications) In our example, SharedFlow is used to model the flow of sensor data:\ninterface SensorSource\u0026lt;out E : SensorEvent\u0026gt; { /** The flow of sensor events. */ val sensorEvents: SharedFlow\u0026lt;E\u0026gt; } Like all flows, they have all the kinds of operators presented in the previous example. Despite this, they do not support, at the moment, all the operators that Rx offers, like groupBy, buffer (in the Rx conception), etc\u0026hellip; (even if some proposals are pending to add them in the framework).\nFor this reason, the consumer of events has been implemented manually, using a loop that, every samplingWindow time, reacts to the data received, updating the state of the system. By the way, if this solution appears to be less elegant, since Flows are, de facto, the porting of Rx\u0026rsquo;s Observable's into the Coroutines world, libraries exists to convert them to Observable and vice versa. This could offer (in some cases and where necessary) a way to use the full power of Rx operator, if needed.\n/** A consumer of sensor events. */ interface SensorSourceConsumer\u0026lt;in E : SensorEvent, out S\u0026gt; { /** The current state of the source consumer. */ val state: S /** Reacts to a sensor event. */ suspend fun react(e: E) } /** A scheduled consumer of sensor events. */ interface ScheduledConsumer\u0026lt;in E : SensorEvent, out S\u0026gt; : SensorSourceConsumer\u0026lt;E, S\u0026gt;, CoroutineScope { /** The interval period. */ val samplingWindow: Duration /** The update logic of the consumer. */ suspend fun update() /** Runs the consumer scheduler. */ fun run() = launch { while (true) { update() delay(samplingWindow) } } } The managers just take care of collecting the data and forwarding it to the appropriate consumer. For example, the ThermostatManager:\nsuspend fun run(sensorSource: Flow\u0026lt;TemperatureEntry\u0026gt;) { thermostat.run() temperatureSensorsChecker.run() sensorSource.collect { thermostat.react(it) temperatureSensorsChecker.react(it) } } Takeaways # Channels in Scala Gears are good to model flow of data that exist without application\u0026rsquo;s request from them: incoming network connections, event streams, etc\u0026hellip; The scheduling mechanism of Task, along with the mutiplexer abstraction, despite having some stability issues, allows to implement flows of hot data which are listened by multiple consumers. Transformation operators inspired by the Reactive world could enhance the expressiveness of the framework, making it more suitable for modeling reactive event-based systems. Previous: Channels as a communication primitive Home "}]
\ No newline at end of file
diff --git a/en.search-data.min.a12762fe0d0ac0594b436c422d039d4c34cf9d6bf009a2124993806d3e50e428.json b/en.search-data.min.a12762fe0d0ac0594b436c422d039d4c34cf9d6bf009a2124993806d3e50e428.json
deleted file mode 100644
index 8fe374f2..00000000
--- a/en.search-data.min.a12762fe0d0ac0594b436c422d039d4c34cf9d6bf009a2124993806d3e50e428.json
+++ /dev/null
@@ -1 +0,0 @@
-[{"id":0,"href":"/direct-style-experiments/docs/01-overview/","title":"01 Overview","section":"Docs","content":" Overview of the project # Context # In the realm of asynchronous programming, the Scala ecosystem offers a set of solid and widely adopted monadic constructs and libraries to tackle complex tasks functionally with elegance and efficiency, like Monix Tasks and Cats Effecs, enabling a wide range of interesting and useful features, like composable error handling, cancellation mechanisms and structured concurrency that the standard library lacks. However, they also come with a cost: the pervasiveness of the flatMap operator to compose values makes the code harder to reason about and difficult and awkward to integrate with regular control structures.\nIn the last years, we have been assisting the increase in adoption of continuation and coroutines in modern runtimes, either exploiting some kind of fibers support, like the project Loom with Virtual Threads, or via code generation, like Kotlin Coroutines, aiming to capture the essence of effects more cleanly compared to monads.\nGoals # The goal of this project is to explore, mainly focusing on Scala, the direct style, developing a few examples (not too complex) leveraging the new strawman library Gears, comparing it with Kotlin Coroutines and the current implementation of monadic Futures, seeking to analyze aspects such as:\nergonomics of the two styles; which of the two approaches has a real advantage in adoption; pros and cons of the two styles; any limitations and difficulties encountered in using them. The contribution # The project is built around three small examples.\nThe first aims, through the implementation of the core of a small Web service, to introduce the basics of asynchronous programming in direct style, focusing on structured concurrency and cancellation.\nThe second introduces the main communication and synchronization primitive of direct style, channels, and how they can be used to exchange data between concurrent tasks (either Futures or coroutines).\nIn this context, a contribution has been made by proposing the extension of the Scala Gears library with the following two abstractions, currently missing, inspired by those of the Kotlin Coroutines:\nterminable channels, i.e. a channel that can be terminated, but whose values can still be read by consumers after its termination until all values are consumed; Flows, modeling a cold stream of asynchronously computed values which are particularly useful for implementing asynchronous functions that produce, not just a single, but a sequence of values. The last example investigates how to implement a reactive-like event-based system using direct style, taking as a use case a small sensor control system in an IoT context that needs to react to events coming from different sensors.\nTo accomplish this, since Scala Gears lacks any kind of transforming operator, some have been introduced on top of channels (taking cues from Reactive frameworks) allowing them to be manipulated functionally.\nConclusions # The current implementation of monadic Futures present in the standard library is insufficient to handle modern asynchronous programming in a structured and safe way: it lacks structured concurrency and cancellation mechanism, is not referential transparent and requires to be ugly mixed with direct style constructs to be used appropriately.\nThis leads to the adoption of effectful libraries that offer these and many other powerful abstractions, beautifully wrapped in monadic constructs. This is the main pro and con of the monadic approach: what makes monads remarkable is their capability to turn statements into programmable values and introduce constructs to transform and compose them functionally in a very elegant (and somehow \u0026ldquo;magical\u0026rdquo; for ones who are not familiar with them) way. Despite this idyllic beauty, monads and effectful libraries require relevant expertise in functional programming to be fully grasped and composed effectively.\nDirect style frameworks are indeed arising as a more natural and intuitive way to handle concurrency, leveraging an imperative-like programming style that is familiar to all developers.\nIn the JVM ecosystem, the most adopted and known direct-style library is the Kotlin Coroutines, which were introduced in 2018 by modifying the Kotlin language (rather than its runtime, like the project Loom has recently done with Virtual Thread) to support suspending functions. The main advantages of Kotlin Coroutines are that they provide suspension and cancellation mechanisms that are simple to understand and use, as well as a good ecosystem for channels and Flows. Despite this, Coroutines are not still perfect: due to their design they partially suffer from the colored functions problem and we need to be aware we can not use the same synchronization concurrency abstractions that we would use in Java with threads (like locks, synchronized, \u0026hellip;) cause they are not designed to be used in the coroutines context.\nScala Gears is an attempt to bring direct style into the Scala ecosystem. Its API design is inspired by Kotlin Coroutines and, despite the fact it achieves suspension, unlike Kotlin, leveraging Virtual Threads for JVM and delimited continuation for Scala Native, the majority of constructs can be mapped to the Kotlin Coroutines ones. Despite being a very young project, it already offers a good set of abstractions for asynchronous programming, although it cannot yet be considered a mature library ready to be used in a production environment:\nsome design choices should be addressed: closing a channel prevents any further reading, precluding the possibility of processing the remaining values (see second example); Task scheduling behavior has a strange behavior with higher-order functions (see third example); missing abstractions: the library is still missing some important abstractions, like the proposed Flows for handling a cold stream of asynchronously computed values and operators for functionally transforming them; performances: the project has been created for experimenting, thus performances have not been considered a priority so far, even though a comparison in overheads of the core primitives has been published. In conclusion, Scala Gears is a promising project that could bring direct-style async programming into the Scala ecosystem, giving, together with boundary and break Scala 3 support, a nice alternative to the current monadic approach, simplifying the way we handle concurrency, making it more natural and intuitive.\nReferences # Scala Gears, Programming Methods Laboratory EPFL Scala 3: What Is \u0026ldquo;Direct Style\u0026rdquo; by D. Wampler Kotlin Coroutines documentation Pre-SIP: Suspended functions and continuations in Scala 3 The Great Concurrency Smackdown: ZIO versus JDK by John A. De Goes Continuaton, coroutine, and generator by A. Ber KotlinConf 2017 - Introduction to Coroutines by Roman Elizarov KotlinConf 2017 - Deep Dive into Coroutines on JVM by Roman Elizarov Kotlin Co-routine Scope by A. Nadiger What Color is your function by B. Nystrom Channel in Kotlin Coroutines SharedFlow and StateFlow Demystifying Kotlin\u0026rsquo;s Channel Flows by S. Cooper Home Next: boundary \u0026amp; break "},{"id":1,"href":"/direct-style-experiments/docs/02-boundaries/","title":"02 Boundaries","section":"Docs","content":" boundary \u0026amp; break # boundary \u0026amp; break Modeling error handling data types with non-local breaks Optional Either + ? boundary \u0026amp; break mechanism provides a cleaner alternative to non-local returns:\nboundary: is short for boundary.apply: the indented code below, passed as body, is a context function that is called within boundary.apply to break, an in-scope given instance of Label is required (i.e. is impossible to break without an enclosing boundary) Users don\u0026rsquo;t define Label instances themselves. Instead, this is done inside the implementation of boundary.apply to provide the capability of doing a non-local return [Ref]: // From the Scala 3 standard library /** Run `body` with freshly generated label as implicit argument. * Catch any breaks associated with that label and return their * results instead of `body`\u0026#39;s result. */ inline def apply[T](inline body: Label[T] ?=\u0026gt; T): T = val local = Label[T]() try body(using local) catch case ex: Break[T] @unchecked =\u0026gt; if ex.label eq local then ex.value else throw ex non-local breaks are implemented as non-fatal exceptions: the implementation is optimized to suppress unnecessary stack traces (which makes exceptions very slow); stack traces are useless since the exceptions are managed rather than exposed to the user abruptly enhanced performance is achieved when a break occurs within the same method, allowing it to be rewritten as a jump call to the enclosing scope within the same stack frame. boundary and break can be particularly useful for error handling (later examples will show some use cases) and inner loops where we need a short exit path. But, most importantly, they lay the foundations (along with a resume mechanism) for building new direct-style concurrency abstractions based on suspensions. Modeling error handling data types with non-local breaks # [Here you can find the full source.]\nIn the following section are presented two data types that can be used to handle errors, both leveraging the boundary and break mechanism. The first (optional) has been presented in the Scalar conference by M. Odersky, while the second has been implemented to apply the same style also to Either data type.\nOptional # /** Represents a computation that will hopefully return * [[Some]]thing or simply [[None]] if it can\u0026#39;t. */ object optional: /** Defines the boundary for an [[Option]] returning computation, * whose [[body]] is given in input. */ inline def apply[T](inline body: Label[None.type] ?=\u0026gt; T): Option[T] = boundary(Some(body)) extension [T](o: Option[T]) /** @return the enclosed [[Option]] object if defined, or break * to the enclosing boundary with [[None]]. */ inline def ?(using label: Label[None.type]): T = o.getOrElse(break(None)) Either + ? # /** Represents a computation that will hopefully return a [[Right]] value but might fail with a [[Left]] one.*/ object either: /** Defines the boundary for the [[Either]] returning computation, whose [[body]] is given in input. */ inline def apply[L, R](inline body: Label[Left[L, Nothing]] ?=\u0026gt; R): Either[L, R] = boundary(Right(body)) /** Quickly break to the enclosing boundary with a [[Left]] filled with [[l]]. */ inline def left[L, R](l: L)(using Label[Left[L, Nothing]]): Either[L, R] = break(Left(l)) extension [L, R](e: Either[L, R]) /** @return this [[Right]] value or break to the enclosing boundary with the [[Left]] value. */ inline def ?(using Label[Left[L, Nothing]]): R = e match case Right(value) =\u0026gt; value case Left(value) =\u0026gt; break(Left(value)) extension [R](t: Try[R]) /** @return this [[Success]] value or break to the enclosing boundary with a [[Left]] containing * the converted `Throwable` exception performed by the implicit [[converter]]. */ inline def ?[L](using Label[Left[L, Nothing]])(using converter: Conversion[Throwable, L]): R = t match case Success(value) =\u0026gt; value case Failure(exception) =\u0026gt; break(Left(converter(exception))) /** An object encapsulating a collection of `Throwable` given converters. */ object EitherConversions: /** Converts a `Throwable` to a `String` with its message. */ given Conversion[Throwable, String] = _.getMessage This kind of data type will be particularly useful in the next examples to quickly break in case of failures, returning the caller a meaningful error message, and simplifying the error-handling code.\nOverview Next: Basic asynchronous constructs "},{"id":2,"href":"/direct-style-experiments/docs/03-basics/","title":"03 Basics","section":"Docs","content":" Basic asynchronous constructs # Basic asynchronous constructs The need for a new Future construct Example: a blog posts service Structure Current monadic Future Direct style: Scala version with gears Kotlin Coroutines Takeaways The need for a new Future construct # The current implementation of the Future monadic construct suffers the following main cons:\nLack of referential transparency; Lack of cancellation mechanisms and structured concurrency; Accidental Sequentiality. To show these weaknesses in practice, a simple example of the core of a web service implementation is presented.\nExample: a blog posts service # Idea: develop a very simple (mocked) service that allows retrieving and storing from a repository blog posts, performing checks on the content and author before the actual storage. The example has been implemented using:\nthe continuation style through the current Scala Future monadic constructs; the direct style, through: the abstractions offered by Gears; Kotlin coroutines. The example is organized into Gradle submodules:\nblog-ws-commons contains code that has been reused for both the monadic and direct versions; blog-ws-monadic contains the monadic Scala style; blog-ws-direct contains the direct version using Scala Gears; blog-ws-direct-kt contains the direct version using Kotlin Coroutines. For this example just the tests are provided. You can explore them in the test folders and run via Gradle using the name of the submodule:\n./gradlew :blog-ws-\u0026lt;monadic | direct | direct-kt\u0026gt;:test Structure # The domain is modeled using abstract data types in a common PostsModel trait:\n/** The model of a simple blog posts service. */ trait PostsModel: /** The post author\u0026#39;s identifier. */ type AuthorId /** The posts title. */ type Title /** The posts body. */ type Body /** The content of the post. */ type PostContent = (Title, Body) /** A post author and their info. */ case class Author(authorId: AuthorId, name: String, surname: String) /** A blog post, comprising an author, title, body and the last modification. */ case class Post(author: Author, title: Title, body: Body, lastModification: Date) /** A function that verifies the content of the post, returning [[Right]] with the content of * the post if the verification succeeds or [[Left]] with the reason why failed. */ type ContentVerifier = (Title, Body) =\u0026gt; Either[String, PostContent] /** A function that verifies the author has appropriate permissions, returning [[Right]] * with their information or [[Left]] with the reason why failed. */ type AuthorsVerifier = AuthorId =\u0026gt; Either[String, Author] To implement the service two components have been conceived, following the Cake Pattern:\nPostsRepositoryComponent exposes the Repository trait allowing to store and retrieve blog posts; mocks a DB technology with an in-memory collection. PostsServiceComponent is the component exposing the Service interface. it could be called by the controller of the ReSTful web service. Both must be designed in an async way.\nCurrent monadic Future # The interface of the repository and services component of the monadic version are presented hereafter and their complete implementation is available here.\n/** The component exposing blog posts repositories. */ trait PostsRepositoryComponent: context: PostsModel =\u0026gt; /** The repository instance. */ val repository: PostsRepository /** The repository in charge of storing and retrieving blog posts. */ trait PostsRepository: /** Save the given [[post]]. */ def save(post: Post)(using ExecutionContext): Future[Post] /** @return a [[Future]] completed with true if a post exists with * the given title, false otherwise. */ def exists(postTitle: Title)(using ExecutionContext): Future[Boolean] /** @return a [[Future]] completed either with a defined optional * post with given [[postTitle]] or an empty one. */ def load(postTitle: Title)(using ExecutionContext): Future[Option[Post]] /** Load the post with the given [[postTitle]]. */ def loadAll()(using ExecutionContext): Future[LazyList[Post]] /** The component blog posts service. */ trait PostsServiceComponent: context: PostsRepositoryComponent \u0026amp; PostsModel =\u0026gt; /** The blog post service instance. */ val service: PostsService /** The service exposing a set of functionalities to interact with blog posts. */ trait PostsService: /** Creates a new blog post with the given [[title]] and [[body]], authored by [[authorId]]. */ def create(authorId: AuthorId, title: Title, body: Body)(using ExecutionContext): Future[Post] /** Get a post from its [[title]]. */ def get(title: Title)(using ExecutionContext): Future[Option[Post]] /** Gets all the stored blog posts in a lazy manner. */ def all()(using ExecutionContext): Future[LazyList[Post]] All the exposed functions, since they are asynchronous, return an instance of Future[T] and require to be called in a scope where a given instance of the ExecutionContext is declared.\nWhat\u0026rsquo;s important to delve into is the implementation of the service, and, more precisely, of the create method. As already mentioned, before saving the post two checks need to be performed:\nthe post author must have permission to publish a post and their information needs to be retrieved (supposing they are managed by another service); the content of the post is analyzed in order to prevent the storage and publication of inappropriate content. Since these operations are independent from each other they can be spawned and run in parallel.\noverride def create(authorId: AuthorId, title: Title, body: Body)(using ExecutionContext): Future[Post] = for exists \u0026lt;- context.repository.exists(title) if !exists post \u0026lt;- save(authorId, title, body) yield post private def save(authorId: AuthorId, title: Title, body: Body)(using ExecutionContext): Future[Post] = val authorAsync = authorBy(authorId) val contentAsync = verifyContent(title, body) for content \u0026lt;- contentAsync author \u0026lt;- authorAsync post = Post(author, content._1, content._2, Date()) _ \u0026lt;- context.repository.save(post) yield post /* Pretending to make a call to the Authorship Service that keeps track of authorized authors. */ private def authorBy(id: AuthorId)(using ExecutionContext): Future[Author] = ??? /* Some local computation that verifies the content of the post is appropriate. */ private def verifyContent(title: Title, body: Body)(using ExecutionContext): Future[PostContent] = ??? This implementation shows the limits of the current monadic Future mechanism:\nif we want to achieve the serialization of futures execution we need to compose them using the flatMap, like in the create function: first, the check on the post existence is performed, and only if it is successful and another post with same title doesn\u0026rsquo;t exist the save function is started\nas a consequence, if we want two futures to run in parallel we have to spawn them before the for-yield, as in the save function. This is error-prone and could lead to unexpected sequentiality, like this:\n// THIS IS WRONG: the two futures are started sequentially! for content \u0026lt;- verifyContent(title, body) author \u0026lt;- authorBy(authorId) post = Post(author, content._1, content._2, Date()) _ \u0026lt;- context.repository.save(post) yield post since the publication of a post can be performed only if both of these checks succeed, it is desirable that, whenever one of the two fails, the other gets canceled. Unfortunately, currently, Scala Futures are not cancellable and provide no structured concurrency mechanism.\nmoreover, they lack referential transparency, i.e. future starts running when they are defined. This means that passing a reference to a future is not the same as passing the referenced expression.\nDirect style: Scala version with gears # The API of the gears library is presented hereafter and is built on top of four main abstractions, three of which are here presented (the fourth in the next example):\nAsync context is \u0026ldquo;a capability that allows a computation to suspend while waiting for the result of an async source\u0026rdquo;. Code that has access to an instance of the Async trait is said to be in an async context and can suspend its execution. Usually, it is provided via given instances. A common way to obtain an Async instance is to use an Async.blocking. Async.Source model an asynchronous source of data that can be polled or awaited by suspending the computation, as well as composed using combinator functions. Futures are the primary (in fact, the only) active elements that encapsulate a control flow that, eventually, will deliver a result (either a computed or a failure value that contains an exception). Since Futures are Async.Sources can be awaited and combined with other Futures, suspending their execution. Tasks are the abstraction created to create delayed Futures, responding to the lack of referential transparency problem. They take the body of a Future as an argument; its run method converts that body to a Future, starting its execution. Promises allow us to define Future\u0026rsquo;s result value externally, instead of executing a specific body of code. classDiagram class Async { \u003c\u003c trait \u003e\u003e +group: CompletionGroup +withGroup(group: CompletionGroup) Async +await[T](src: Async.Source[T]) T +current() Async$ +blocking[T](body: Async ?=\u003e T) T$ +group[T](body: Async ?=\u003e T) T$ } class `Async.Source[+T]` { \u003c\u003c trait \u003e\u003e +poll(k: Listener[T]) Boolean +poll() Option[T] +onComplete(k: Listener[T]) +dropListener(k: Listener[T]) +awaitResult()(using Async) T } Async *--\u003e `Async.Source[+T]` class OriginalSource { \u003c\u003c abstract class \u003e\u003e } `Async.Source[+T]` \u003c|-- OriginalSource class `Listener[-T]` { \u003c\u003c trait \u003e\u003e +lock: Listener.ListenerLock | Null +complete(data: T, source: Async.Source[T]) +completeNow(data: T, source: Async.Source[T]) Boolean +apply[T](consumer: (T, Source[T]) =\u003e Unit) Listener[T]$ } `Async.Source[+T]` *--\u003e `Listener[-T]` class `Future[+T]` { \u003c\u003c trait \u003e\u003e +apply[T](body: Async ?=\u003e T) Future[T]$ +now[T](result: Try[T]) Future[T] +zip[U](f2: Future[U]) Future[T, U] +alt(f2: Future[T]) Future[T] +altWithCancel(f2: Future[T]) Future[T] } class `Promise[+T]` { \u003c\u003c trait \u003e\u003e +apply() Promise[T]$ +asFuture Future[T] +complete(result: Try[T]) } OriginalSource \u003c|-- `Future[+T]` `Future[+T]` \u003c|-- `Promise[+T]` class `Task[+T]` { +apply(body: (Async, AsyncOperations) ?=\u003e T) Task[T]$ +run(using Async, AsyncOperations) Future[+T] } `Future[+T]` \u003c--* `Task[+T]` class Cancellable { \u003c\u003c trait \u003e\u003e +group: CompletionGroup +cancel() +link(group: CompletionGroup) +unlink() } Cancellable \u003c|-- `Future[+T]` class Tracking { \u003c\u003c trait \u003e\u003e +isCancelled Boolean } Cancellable \u003c|-- Tracking class CompletionGroup { +add(member: Cancellable) +drop(member: Cancellable) } Tracking \u003c|-- CompletionGroup Async *--\u003e CompletionGroup Going back to our example, the interface of both the repository and service components becomes (here you can find the complete sources):\n/** The component exposing blog posts repositories. */ trait PostsRepositoryComponent: context: PostsModel =\u0026gt; /** The repository instance. */ val repository: PostsRepository /** The repository in charge of storing and retrieving blog posts. */ trait PostsRepository: /** Save the given [[post]]. */ def save(post: Post)(using Async): Either[String, Post] /** Return true if a post exists with the given title, false otherwise. */ def exists(postTitle: Title)(using Async): Either[String, Boolean] /** Load the post with the given [[postTitle]]. */ def load(postTitle: Title)(using Async): Either[String, Option[Post]] /** Load all the saved post. */ def loadAll()(using Async): Either[String, LazyList[Post]] /** The blog posts service component. */ trait PostsServiceComponent: context: PostsRepositoryComponent \u0026amp; PostsModel =\u0026gt; /** The blog post service instance. */ val service: PostsService /** The service exposing a set of functionalities to interact with blog posts. */ trait PostsService: /** Creates a new blog post with the given [[title]] and [[body]], authored by [[authorId]], * or a string explaining the reason of the failure. */ def create(authorId: AuthorId, title: Title, body: Body)(using Async): Either[String, Post] /** Get a post from its [[title]] or a string explaining the reason of the failure. */ def get(title: Title)(using Async): Either[String, Option[Post]] /** Gets all the stored blog posts in a lazy manner or a string explaining the reason of the failure. */ def all()(using Async): Either[String, LazyList[Post]] As you can see, Futures are gone and the return type it\u0026rsquo;s just the result of their intent (expressed with Either to return a meaningful message in case of failure). The fact they are suspendable is expressed using the Async context, which is required to invoke those functions.\nKey inspiring principle (actually, taken by Kotlin)\n❝Concurrency is hard! Concurrency has to be explicit!❞\nBy default the code is serial. If you want to opt-in concurrency you have to explicitly use a Future or Task spawning a new control flow that executes asynchronously, allowing the caller to continue its execution.\nThe other important key feature of the library is the support for structured concurrency and cancellation mechanisms:\nFutures are Cancellable instances;\nWhen you cancel a future using the cancel() method, it promptly sets its value to Failure(CancellationException). Additionally, if it\u0026rsquo;s a runnable future, the thread associated with it is interrupted using Thread.interrupt().\nto avoid immediate cancellation, deferring the cancellation after some block, is possible using uninterruptible function:\nval f = Future: // this can be interrupted uninterruptible: // this cannot be interrupted *immediately* // this can be interrupted Futures are nestable; the lifetime of nested computations is contained within the lifetime of enclosing ones. This is achieved using CompletionGroups, which are cancellable objects themselves and serve as containers for other cancellable objects; once they are canceled, all of their members are canceled as well.\nThe group is accessible through Async.current.group; A cancellable object can be included inside the cancellation group of the async context using the link method; this is what the implementation of the Future does, under the hood. The implementation of the create function with direct style in Gears looks like this:\noverride def create(authorId: AuthorId, title: Title, body: Body)(using Async): Either[String, Post] = either: if context.repository.exists(title).? then left(s\u0026#34;A post entitled $title already exists\u0026#34;) val f = Future: val content = verifyContent(title, body).run val author = authorBy(authorId).run content.zip(author).await val (post, author) = f.awaitResult.? context.repository.save(Post(author.?, post.?._1, post.?._2, Date())).? /* Pretending to make a call to the Authorship Service that keeps track of authorized authors. */ private def authorBy(id: AuthorId): Task[Either[String, Author]] = ??? /* Some local computation that verifies the content of the post is appropriate. */ private def verifyContent(title: Title, body: Body): Task[Either[String, PostContent]] = ??? Some remarks:\nthe either boundary have been used to quickly return a Right[String, Post] object in case something goes wrong; authorBy and verifyContent returns referential transparent Task instances. Running them spawns a new Future instance; Thanks to structured concurrency and zip combinator we can obtain that if one of the nested two futures fails the enclosing future is cancelled, cancelling also all its unterminated children zip: combinator function returning a pair with the results if both Futures succeed, otherwise fail with the failure that was returned first.\nBe aware of the fact to achieve cancellation is necessary to enclose both the content verification and authorization task inside an enclosing Future, since the zip doesn\u0026rsquo;t provide a cancellation mechanism per se. The following code wouldn\u0026rsquo;t work as expected!\n// WRONG: doesn\u0026#39;t provide cancellation! val contentVerifier = verifyContent(title, body).run val authorizer = authorBy(authorId).run val (post, author) = contentVerifier.zip(authorizer).awaitResult.? 👉🏻 To showcase the structured concurrency and cancellation mechanisms of Scala Gears tests have been prepared:\nStructuredConcurrencyTest CancellationTest Other combinator methods, available on Futures instance:\nCombinator Goal Future[T].zip(Future[U]) Parallel composition of two futures. If both futures succeed, succeed with their values in a pair. Otherwise, fail with the failure that was returned first Future[T].alt(Future[T]) / Seq[Future[T]].altAll Alternative parallel composition. If either task succeeds, succeed with the success that was returned first. Otherwise, fail with the failure that was returned last (race all futures). Future[T].altWithCancel(Future[T]) / Seq[Future[T]].altAllWithCancel Like alt but the slower future is cancelled. Seq[Future[T]].awaitAll .await for all futures in the sequence, returns the results in a sequence, or throws if any futures fail. Seq[Future[T]].awaitAllOrCancel Like awaitAll, but cancels all futures as soon as one of them fails. Kotlin Coroutines # A coroutine in Kotlin is an instance of a suspendable computation.\nTheir API is quite similar to the Scala Gears, which has taken inspiration from Kotlin coroutines. To try to make a comparison, the following table shows the correspondence between the two libraries:\nScala Gears Kotlin Coroutines Async CoroutineScope Future[Unit] Job Future[T] Deferred\u0026lt;T\u0026gt; def all()(using Async) suspend fun all() In the Kotlin Coroutine library, CoroutineScope is an interface that defines a single property, coroutineContext, which returns the CoroutineContext that defines the scope in which the coroutine runs.\npublic interface CoroutineScope { /** Returns the context of this scope. */ public val coroutineContext: CoroutineContext } Every coroutine must be executed in a coroutine context, which is a collection of key-value pairs that provide contextual information for the coroutine, including a dispatcher, that determines what thread or threads the coroutine uses for its execution, and the Job of the coroutine, which represents a cancellable background piece of work with a life cycle that culminates in its completion.\nDifferent ways to create a scope:\nGlobalScope.launch launching a new coroutine in the global scope \u0026ndash; discouraged because it can lead to memory leaks; CoroutineScope(Dispatchers.Default), using the constructor with a dispatcher; runBlocking - equivalent to the Gears Async.blocking - provides a way to run a coroutine in the MainScope, i.e. on the main/UI thread. Useful dispatchers:\nDefault dispatcher: to run CPU-intensive functions. If we forget to choose our dispatcher, this dispatcher will be selected by default; IO dispatcher: to run I-O bound computation, where we block waiting for input-output operations to complete, like network-related operations, file operations, etc.; Unconfined dispatcher: it isn\u0026rsquo;t restricted to a particular thread, i.e. doesn\u0026rsquo;t change the thread of the coroutine, it operates on the same thread where it was initiated; Main dispatcher: used when we want to interact with the UI. It is restricted to the main thread. Several coroutine builders exist, like launch, async, withContext which accept an optional CoroutineContext parameter that can be used to specify the dispatcher and other context elements.\nfun main(): Unit = runBlocking { // this: CoroutineScope val job: Job = this.launch(Dispatchers.IO) { // launch a new coroutine and continue delay(1000L) // non-blocking delay for 1 second (default time unit is ms) print(\u0026#34;Kotlin Coroutine!\u0026#34;) // print after delay } val job2: Deferred\u0026lt;String\u0026gt; = async { delay(100L) \u0026#34;Hello\u0026#34; } print(job2.await() + \u0026#34; \u0026#34;) // wait until child coroutine completes job.join() // wait until the job is done and \u0026#34;Kotlin Coroutine!\u0026#34; is printed } suspending functions are marked with the suspend keyword; they can use other suspending functions to suspend the execution of a coroutine.\nCoroutines follow the principle of structured concurrency: coroutines can be arranged into parent-child hierarchies where the cancellation of a parent leads to the immediate cancellation of all its children recursively. Failure of a child with an exception immediately cancels its parent and, consequently, all its other children.\nGoing back to our example, the interface of the service with Kotlin coroutines looks like this (here you can find the complete sources):\n/** The service exposing a set of functionalities to interact with blog posts. */ interface PostsService { /** Creates a new post. */ suspend fun create(authorId: String, title: String, body: String): Result\u0026lt;Post\u0026gt; /** Retrieves a post by its title. */ suspend fun get(title: String): Result\u0026lt;Post\u0026gt; /** Retrieves all the posts. */ suspend fun getAll(): Result\u0026lt;Sequence\u0026lt;Post\u0026gt;\u0026gt; } The implementation of the create function:\noverride suspend fun create(authorId: String, title: String, body: String): Result\u0026lt;Post\u0026gt; = runCatching { coroutineScope { require(!repository.exists(title)) { \u0026#34;Post with title $title already exists\u0026#34; } val content = async { verifyContent(title, body) } val author = async { authorBy(authorId) } val post = Post(author.await(), content.await(), Date()) repository.save(post) } } /* Pretending to make a call to the Authorship Service that keeps track of authorized authors. */ private suspend fun authorBy(id: String): Author { ... } /* Some local computation that verifies the content of the post is appropriate. */ private suspend fun verifyContent(title: String, body: String): PostContent { ... } a coroutineScope is a suspending function used to create a new coroutine scope: it suspends the execution of the current coroutine, releasing the underlying thread for other usages; As we said previously, the failure of a child with an exception immediately cancels its parent and, consequently, all its other children: this means that, for handling the cancellation of nested coroutines, we don\u0026rsquo;t need to do anything special with coroutineScope no matter the order in which coroutines are awaited, if one of them fails with an exception it is propagated upwards, cancelling all other ones this is not the case for supervisorScope, a coroutine builder ensuring that child coroutines can fail independently without affecting the parent coroutine. have a look to this test This is an advantage over the Scala Gears, where operators like zip and altWithCancel are necessary! Takeaways # Scala Gears offers, despite the syntactical differences, very similar concepts to Kotlin Coroutines, with structured concurrency and cancellation mechanisms; Kotlin Coroutines handles the cancellation of nested coroutines more easily than Scala Gears, where special attention is required; As stated by M. Odersky the Async capability is better than suspend because let defines functions that work for synchronous as well as asynchronous function arguments without changing anything, while in Kotlin suspendable functions passed as an argument in higher-order functions must be tagged with suspend keyword. Previous: boundary \u0026amp; break Next: Channels as a communication primitive "},{"id":3,"href":"/direct-style-experiments/docs/04-channels/","title":"04 Channels","section":"Docs","content":" Channels as a communication primitive # Channels as a communication primitive Introduction Analyzer example Future monadic version Scala Gears version Kotlin Coroutines version Introducing Flows in Gears Showcasing Flows Takeaways Introduction # The fourth, yet not mentioned, abstraction of both Kotlin Coroutines and Scala Gears is the channel. Channels represent the primitive communication and coordination means to exchange Future (or coroutines in the case of Kotlin) results. They are, at least conceptually, very similar to a queue where it is possible to send (and receive) data \u0026ndash; basically, exploiting the producer-consumer pattern.\nclassDiagram class `SendableChannel[-T]` { \u003c\u003c trait \u003e\u003e +sendSource(x: T) Async.Source[Either[Closed, Unit]] +send(x: T)(using Async) Unit } class `ReadableChannel[+T]` { \u003c\u003c trait \u003e\u003e +readSource Async.Source[Either[Closed, T]] +read()(using Async) Either[Closed, T] } class `Channel[T]` { \u003c\u003c trait \u003e\u003e +asSendable: SendableChannel[T] +asReadable: ReadableChannel[T] +asCloseable: java.io.Closeable } namespace java io { class Closeable { \u003c\u003c interface \u003e\u003e +close() } } `SendableChannel[-T]` \u003c|-- `Channel[T]` Closeable \u003c|-- `Channel[T]` `ReadableChannel[+T]` \u003c|-- `Channel[T]` The channel is defined through three distinct interfaces: SendableChannel[-T], ReadableChannel[+T] and Channel[T], where the latter extends from both SendableChannel and ReadableChannel. Typically, a Channel is created and a SendableChannel and ReadableChannel instances are respectively provided to the producer and the consumer, restricting their access to it. The same, almost identical, design is present also in Kotlin Coroutines where SendChannel and ReceiveChannel take over, respectively, the Gears SendableChannel and ReadableChannel.\nChannel inherits from java.io.Closable, making them closable objects: once closed, they raise ChannelClosedException when attempting to write to them and immediately return a Left(Closed) when attempting to read from them, preventing the consumer from finishing reading all the values sent on the channel before its closing. This is not the case for Kotlin Coroutines where closing a channel indicates that no more values are coming, but doesn\u0026rsquo;t prevent consuming already sent values. Moreover, in Kotlin is possible to use a regular for loop to receive elements from a channel (blocking the coroutine): val channel = Channel\u0026lt;Int\u0026gt;() launch { for (x in 1..5) channel.send(x * x) channel.close() // we\u0026#39;re done sending } for (y in channel) println(y) // blocks until channel is closed println(\u0026#34;Done!\u0026#34;) Similar behavior can be achieved also in Gears extending the framework with the concept of Terminable channel. After all, closing a channel in coroutines is a matter of sending a special token to it, allowing stop the iteration as soon as this token is received. [The full implementation can be found in commons submodule, pimping package.]\n/** A token to be sent to a channel to signal that it has been terminated. */ case object Terminated type Terminated = Terminated.type /** A union type of [[T]] and [[Terminated]]. */ type Terminable[T] = T | Terminated /** Exception being raised by [[TerminableChannel.send()]] on terminated [[TerminableChannel]]. */ class ChannelTerminatedException extends Exception /** A [[Channel]] that can be terminated, signalling no more items will be sent, * still allowing to consumer to read pending values. * Trying to `send` values after its termination arise a [[ChannelTerminatedException]]. * When one consumer reads the [[Terminated]] token, the channel is closed. Any subsequent * read will return `Left(Channel.Closed`. */ trait TerminableChannel[T] extends Channel[Terminable[T]]: def terminate()(using Async): Unit object TerminableChannel: /** Creates a [[TerminableChannel]] backed to [[SyncChannel]]. */ def ofSync[T]: TerminableChannel[T] = TerminableChannelImpl(SyncChannel()) /** Creates a [[TerminableChannel]] backed to [[BufferedChannel]]. */ def ofBuffered[T]: TerminableChannel[T] = TerminableChannelImpl(BufferedChannel()) /** Creates a [[TerminableChannel]] backed to an [[UnboundedChannel]]. */ def ofUnbounded[T]: TerminableChannel[T] = TerminableChannelImpl(UnboundedChannel()) private class TerminableChannelImpl[T](c: Channel[Terminable[T]]) extends TerminableChannel[T]: opaque type Res[R] = Either[Channel.Closed, R] private var _terminated: Boolean = false override val readSource: Async.Source[Res[Terminable[T]]] = c.readSource.transformValuesWith { case Right(Terminated) =\u0026gt; c.close(); Left(Channel.Closed) case v @ _ =\u0026gt; v } override def sendSource(x: Terminable[T]): Async.Source[Res[Unit]] = synchronized: if _terminated then throw ChannelTerminatedException() else if x == Terminated then _terminated = true c.sendSource(x) override def close(): Unit = c.close() override def terminate()(using Async): Unit = try send(Terminated) // It happens only at the close of the channel due to the call (inside Gears library) of // a CellBuf.dequeue(channels.scala:239) which is empty! catch case _: NoSuchElementException =\u0026gt; () // e.printStackTrace() Now, thanks to this extension, also in Scala Gears is possible to write:\nval channel = TerminableChannel.ofUnbounded[Int] Future: (0 until 10).foreach(channel.send(_)) channel.terminate() // we\u0026#39;re done sending channel.foreach(println(_)) // blocks until channel is closed println(\u0026#34;Done!\u0026#34;) [Other tests can be found in TerminableChannelTest.]\nOn top of this new abstraction is possible to implement, for example, the foreach and toSeq methods, which can be useful to wait for all the items sent over the channel.\nobject TerminableChannelOps: extension [T: ClassTag](c: TerminableChannel[T]) /** Blocking consume channel items, executing the given function [[f]] for each element. */ @tailrec def foreach[U](f: T =\u0026gt; U)(using Async): Unit = c.read() match case Left(Channel.Closed) =\u0026gt; () case Right(value) =\u0026gt; value match case Terminated =\u0026gt; () case v: T =\u0026gt; f(v); foreach(f) /** @return a [[Seq]] containing channel items, after having them read. * This is a blocking operation! */ def toSeq(using Async): Seq[T] = var results = Seq[T]() c.foreach(t =\u0026gt; results = results :+ t) results Three types of channels exist:\nSynchronous Channels: links a read request with a send within a rendezvous send (read) suspend the process until a consumer read (send) the value; in Kotlin they are called Rendezvous Channels. Buffered Channels: a version of a channel with an internal buffer of fixed size send suspend the producer process if it is full; otherwise, it appends the value to the buffer, returning immediately; read suspend if the channel is empty, waiting for a new value. Unbounded Channels: a version of a channel with an unbounded buffer if the programs run out of memory you can get an out-of-memory exception! in Kotlin they are called Unlimited Channel. Kotlin offers also a fourth type: the Conflated Channel, where every new element sent to it overwrites the previously sent one, never blocking, so that the receiver gets always the latest element.\nConcerning channel behavior, it is important to note that:\nMultiple producers can send data to the channel, as well as multiple consumers can read them, but each element is handled only once, by one of them, i.e. consumers compete with each other for sent values; Once the element is handled, it is immediately removed from the channel; Channels are fair: send and read operations to channels are fair w.r.t. the order of their invocations from multiple threads (they are served in first-in first-out order). Analyzer example # To show channels in action an example has been prepared:\nIdea: we want to realize a little asynchronous library allowing clients to collect the common statistics about repositories (issues, stars, last release) and contributors of a given GitHub organization.\nThe final result is a GUI application that, given an organization name, starts the analysis of all its repositories, listing their information along with all their contributors as soon as they are computed. Moreover, the application allows the user to cancel the current computation at any point in time.\n[This example has been inspired by this tutorial.]\nTo start the application:\n./gradlew analyzer-\u0026lt;direct | direct-kt | monadic\u0026gt;:run In order to run the application you need to place inside the analyzer-commons directory a .env file containing your personal GitHub access token, like:\nGH_TOKEN=.... or having set an environment variable named GH_TOKEN.\nThe example is structured in two different packages: lib and client. The former contains the logic of the library, while the latter contains the application (client code). As usual, it has been implemented using monadic Futures, as well as using Scala Gears and Kotlin Coroutines.\nFuture monadic version # [The sources are available inside the analyzer-monadic submodule.]\nThe entry point of the library is the Analyzer interface which takes in input the organization name and a function through which is possible to react to results while they are computed.\nSince we want to achieve cancellation, the monadic version leverages Monix Task, which is returned by the analyze method wrapped in an EitherT monad transformer to allow handling errors functionally.\n/** A generic analyzer of organization/group/workspace repositories. */ trait Analyzer: /** @return a [[EitherT]] encapsulating a [[Task]] that performs the analysis of the * [[organizationName]]\u0026#39;s repositories, providing the results incrementally to the * [[updateResults]] function. */ def analyze(organizationName: String)( updateResult: RepositoryReport =\u0026gt; Unit, ): EitherT[Task, String, Seq[RepositoryReport]] To retrieve data from GitHub, a RepositoryService interface has been created, following the same pattern:\n/** A service exposing functions to retrieve data from a central hosting repository service. */ trait RepositoryService: /** @return a [[EitherT]] encapsulating a [[Task]] which get all the repositories * owned by [[organizationName]]. */ def repositoriesOf(organizationName: String): EitherT[Task, String, Seq[Repository]] /** @return a [[EitherT]] encapsulating a [[Task]] which get all the contributors * of [[repositoryName]] owned by [[organizationName]]. */ def contributorsOf(organizationName: String, repositoryName: String): EitherT[Task, String, Seq[Contribution]] /** @return a [[EitherT]] encapsulating a [[Task]] which get the last release * of [[repositoryName]] owned by [[organizationName]]. */ def lastReleaseOf(organizationName: String, repositoryName: String): EitherT[Task, String, Release] The implementation of the Analyzer is shown in the following code snippet and performs the following steps:\nfirst, the list of repositories is retrieved; if no error occurs, the analysis of each repository is performed concurrently, thanks to the parTraverse operator; the analysis of each repository consists of retrieving the contributors and the last release of the repository and then updating the result through the updateResult function. Since both the contributors and last release retrieval are independent of each other, they are performed concurrently, thanks to Task.parZip2. override def analyze(organizationName: String)( updateResult: RepositoryReport =\u0026gt; Unit, ): EitherT[Task, String, Seq[RepositoryReport]] = for repositories \u0026lt;- repositoryService.repositoriesOf(organizationName) reports \u0026lt;- repositories.parTraverse(r =\u0026gt; EitherT.right(r.performAnalysis(updateResult))) yield reports extension (r: Repository) private def performAnalysis(updateResult: RepositoryReport =\u0026gt; Unit): Task[RepositoryReport] = val contributorsTask = repositoryService.contributorsOf(r.organization, r.name).value val releaseTask = repositoryService.lastReleaseOf(r.organization, r.name).value for result \u0026lt;- Task.parZip2(contributorsTask, releaseTask) report = RepositoryReport(r.name, r.issues, r.stars, result._1.getOrElse(Seq.empty), result._2.toOption) _ \u0026lt;- Task(updateResult(report)) yield report Client-side, when a new session is requested, the Analyzer is used to start the computation, during which the single reports are aggregated and the UI is updated. Whenever desired, the current computation can be stopped by canceling the Monix CancelableFuture returned by the runToFuture method, through which the returned Task from the Analyzer is started.\nclass MonadicAppController extends AppController: import monix.execution.Scheduler.Implicits.global private val view = AnalyzerView.gui(this) private val analyzer = Analyzer(RepositoryService.ofGitHub) private var currentComputation: Option[CancelableFuture[Unit]] = None view.run() override def runSession(organizationName: String): Unit = var organizationReport: OrganizationReport = (Map(), Set()) val f = analyzer.analyze(organizationName) { report =\u0026gt; organizationReport = organizationReport.mergeWith(report) view.update(organizationReport) }.value.runToFuture.map { case Left(value) =\u0026gt; view.error(value); case Right(_) =\u0026gt; view.endComputation() } currentComputation = Some(f) override def stopSession(): Unit = currentComputation foreach (_.cancel()) Scala Gears version # [The sources are available inside the analyzer-direct submodule.]\nThe interfaces of the Direct Style with Gears differ from the monadic one by their return type, which is a simpler Either data type, and by the fact they are suspendable functions, hence they require an Async context to be executed. This is the first important difference: the analyze method, differently from the monadic version, doesn\u0026rsquo;t return immediately the control; instead, it suspends the execution of the client until the result is available (though offering the opportunity to react to each update). This obeys the principle of explicit asynchrony: if the client wants to perform this operation asynchronously, it has to opt in explicitly, using a Future.\n/** A generic analyzer of organization/group/workspace repositories. */ trait Analyzer: /** Performs a **suspending** analysis of the [[organizationName]]\u0026#39;s repositories, * providing the results incrementally to the [[updateResults]] function. * @return [[Right]] with the overall results of the analysis or [[Left]] with * an error message in case of failure. */ def analyze(organizationName: String)( updateResults: RepositoryReport =\u0026gt; Unit, )(using Async, AsyncOperations): Either[String, Seq[RepositoryReport]] /** A service exposing functions to retrieve data from a central hosting repository service. */ trait RepositoryService: /** @return [[Right]] with the [[Seq]]uence of [[Repository]] owned by the given * [[organizationName]] or a [[Left]] with a explanatory message in case of errors. */ def repositoriesOf(organizationName: String)(using Async): Either[String, Seq[Repository]] /** @return [[Right]] with the [[Seq]]uence of [[Contribution]] for the given * [[repositoryName]] owned by the given [[organizationName]] or a [[Left]] with an * explanatory message in case of errors. */ def contributorsOf( organizationName: String, repositoryName: String )(using Async): Either[String, Seq[Contribution]] /** @return a [[Right]] with the last [[Release]] of the given [[repositoryName]] owned * by [[organizationName]] if it exists, or a [[Left]] with a explanatory message * in case of errors. */ def lastReleaseOf( organizationName: String, repositoryName: String )(using Async): Either[String, Release] The implementation of the Analyzer leverages Channels to perform the concurrent analysis of the repositories:\noverride def analyze(organizationName: String)( updateResults: RepositoryReport =\u0026gt; Unit, )(using Async, AsyncOperations): Either[String, Seq[RepositoryReport]] = either: val reposInfo = repositoryService.repositoriesOf(organizationName).? // 1 .map(_.performAnalysis.run) // 2 val collector = Collector[RepositoryReport](reposInfo.toList*) // 3 reposInfo.foreach(_ =\u0026gt; updateResults(collector.results.read().asTry.?.awaitResult.?) // 4 ) reposInfo.awaitAll // 5 extension (r: Repository) protected def performAnalysis(using Async): Task[RepositoryReport] = Task: val contributions = Future: // concurrent! repositoryService.contributorsOf(r.organization, r.name) val release = repositoryService.lastReleaseOf(r.organization, r.name) RepositoryReport(r.name, r.issues, r.stars, contributions.await.getOrElse(Seq()), release.toOption) first, we get all the repositories of the requested organization for each of them, the contributors and the last release are retrieved concurrently, starting a Future Future results are gathered inside a Collector allowing to collect a list of futures into a channel of futures, arriving as they finish. the retrieval of the contributors and the last release are performed in parallel read results from the channel as they come, calling the updateResult reaction function. Overall results are returned to the client. Although it works, the proposed solution suffers from a performance issue when the organization we want to analyze has a large number of repositories. Indeed, the GitHub API, like many ReSTful APIs, implements pagination: if the response includes many results, they are paginated, returning a subset of them; it is the responsibility of the client to request more data (pages). Until now, the RepositoryService has been implemented to return the whole results in one shot, leading to suspension until all pages are retrieved. It would be desirable, instead, to start performing the analysis as soon as one page is obtained from the API.\nTo do so, the interface of the RepositoryService has been extended with new methods, incremental***, returning a TerminableChannel of results:\ntrait RepositoryService: /** @return a [[Terminable]] [[ReadableChannel]] with the [[Repository]] owned by the given * [[organizationName]], wrapped inside a [[Either]] for errors management. */ def incrementalRepositoriesOf( organizationName: String, )(using Async): TerminableChannel[Either[String, Repository]] /** @return a [[Terminable]] [[ReadableChannel]] with the [[Contribution]] made by users to the * given [[repositoryName]] owned by [[organizationName]], wrapped inside a [[Either]]. */ def incrementalContributorsOf( organizationName: String, repositoryName: String, )(using Async): TerminableChannel[Either[String, Contribution]] // ... Then, the implementation of the analyze method becomes:\noverride def analyze(organizationName: String)( updateResults: RepositoryReport =\u0026gt; Unit, )(using Async, AsyncOperations): Either[String, Seq[RepositoryReport]] = either: val reposInfo = repositoryService.incrementalRepositoriesOf(organizationName) // 1 var futures = Seq[Future[RepositoryReport]]() reposInfo.foreach { repository =\u0026gt; // 2 futures = futures :+ Future: // 3 val report = repository.?.performAnalysis.run.awaitResult.? updateResults(report) report } futures.awaitAllOrCancel we get the channel of repositories from the repository service; the foreach method of TerminableChannel is used to iterate over all the repositories sent over the channel as soon as they are retrieved by the service. This is a blocking operation, i.e. it suspends until all the repositories are retrieved; we start the analysis in a separate Future (i.e. thread): this allows you to start the analysis as soon as a repository is fetched by the channel, preventing starting the analysis of the next repository only when the previous one is finished; once all the repositories are retrieved, i.e. the foreach terminates, we wait for the completion of all the started Futures. Indeed, when the foreach terminates, we have the guarantee that all started futures have been started, but not yet completed! Kotlin Coroutines version # [The sources are available inside the analyzer-direct-kt submodule.]\nThe analyzer interface reflects the Scala Gears one: a Result is used in place of Either, and the suspendable function udateResults is marked with the suspend keyword in place of the using Async context.\ninterface Analyzer { suspend fun analyze( organizationName: String, updateResults: suspend (RepositoryReport) -\u0026gt; Unit = { _ -\u0026gt; }, ): Result\u0026lt;Set\u0026lt;RepositoryReport\u0026gt;\u0026gt; } Its channel-based implementation, despite syntactic differences, is also very similar to that of Scala Gears, at least conceptually:\nwe get all the repositories; for each of them, an analysis is started to retrieve the contributors and the last release; each analysis is started in a separate coroutine whose results are sent to a channel; as usual, the contributors and the last release are retrieved concurrently, using the async coroutine builder; results are aggregated as they come from the channel. override suspend fun analyze( organizationName: String, updateResults: suspend (RepositoryReport) -\u0026gt; Unit, ): Result\u0026lt;Set\u0026lt;RepositoryReport\u0026gt;\u0026gt; = coroutineScope { runCatching { val repositories = provider.repositoriesOf(organizationName).getOrThrow() // 1 val resultsChannel = analyzeAll(organizationName, repositories) // 2 collectResults(resultsChannel, repositories.size, updateResults) // 3 } } private fun CoroutineScope.analyzeAll(organizationName: String, repositories: List\u0026lt;Repository\u0026gt;) = Channel\u0026lt;RepositoryReport\u0026gt;().also { repositories.map { r -\u0026gt; launch { // a new coroutine for each repository is started val contributors = async { provider.contributorsOf(organizationName, r.name).getOrThrow() } val release = provider.lastReleaseOf(organizationName, r.name).getOrThrow() it.send(RepositoryReport(r.name, r.issues, r.stars, contributors.await(), release)) } } } private suspend fun collectResults( resultsChannel: Channel\u0026lt;RepositoryReport\u0026gt;, expectedResults: Int, updateResults: suspend (RepositoryReport) -\u0026gt; Unit, ) = mutableSetOf\u0026lt;RepositoryReport\u0026gt;().apply { repeat(expectedResults) { val report = resultsChannel.receive() add(report) updateResults(report) } resultsChannel.close() } Where, instead, Kotlin Coroutines shine is the implementation of the RepositoryService for supporting incremental retrieval of repositories and contributors.\nIndeed, Kotlin has a built-in support for cold streams, called Flow. They are very similar (actually they have been inspired to) cold observable in reactive programming, and they are the perfect fit for functions that need to return a stream of asynchronously computed values.\nThe RepositoryService has been here extended with new methods, flowing***, returning a Flow of results:\nclass GitHubRepositoryProvider { fun flowingRepositoriesOf(organizationName: String): Flow\u0026lt;List\u0026lt;Repository\u0026gt;\u0026gt; fun flowingContributorsOf(organizationName: String, repositoryName: String): Flow\u0026lt;List\u0026lt;Contribution\u0026gt;\u0026gt; } As already mentioned, the Flow is a cold stream, meaning that it is not started until it is collected. Once the collect method is called a new stream is created and data starts to \u0026ldquo;flow\u0026rdquo;.\nThey offer several useful operators for transforming and combining them functionally (not a complete list):\nIntermediate flow operators:\nfilter/filterNot to filter out unwanted values; map to transform the values; transform to implement more complex transformations (possibly involving suspending operations); take and its variant (e.g. takeWhile) to limit the number of values emitted; onEach to perform side-effects for each value emitted. Terminal flow operators:\nconversions to various collection types, like toList, toSet; first, last, single to retrieve the first, last or single value emitted; reduce to perform some kind of operation over all items, reducing them to a single one; fold to perform some kind of operation over all items, starting from an initial value, accumulating a result. Flows combining operators:\nmerge to combine multiple flows into a single one, emitting values from all of them; zip combines the corresponding values of two flows; flatMapConcat / flatMapMerge to transform each value into a flow and then concatenate/merge them. Moreover, like in Rx:\nit is possible to control the context in which the flow is executed using the flowOn operator, which changes the context for all the steps above it (so it is typically used as the last step in a function); some backpressure strategies are supported, which are used to handle the situation when the producer is emitting values faster than the consumer can process them. In Kotlin Coroutines, the backpressure is managed by the buffer (and its variant, like conflated, sample, debounce) operator, which allows to buffer a certain number of values before the consumer starts to process them. override suspend fun analyze( organizationName: String, updateResults: suspend (RepositoryReport) -\u0026gt; Unit, ): Result\u0026lt;Set\u0026lt;RepositoryReport\u0026gt;\u0026gt; = coroutineScope { runCatching { val reports = provider.flowingRepositoriesOf(organizationName) .flatMapConcat { analyzeAll(it) } .flowOn(Dispatchers.Default) var allReports = emptySet\u0026lt;RepositoryReport\u0026gt;() // until here just \u0026#34;configuration\u0026#34; reports.collect { updateResults(it) allReports = allReports + it } allReports } } Concerning flows, an important thing to note is that they are just asynchronous generators that run some suspending code when you collect them. Thus, per se, they don\u0026rsquo;t introduce new coroutines or concurrency mechanism. To achieve concurrency, for example emitting values concurrently by multiple coroutines, is necessary to use a channelFlow, a pre-cooked way to inject a coroutineContext and a Channel through which is possible to pass the values to be emitted from a background coroutines back to the main control flow:\nfun analyzeAll(repositories: List\u0026lt;Repository\u0026gt;): Flow\u0026lt;RepositoryReport\u0026gt; = channelFlow { repositories.forEach { repository -\u0026gt; launch { val release = async { provider.lastReleaseOf(repository.organization, repository.name).getOrThrow() } provider.flowingContributorsOf(repository.organization, repository.name).toList().forEach { // emit this value send(RepositoryReport(repository.name, repository.issues, repository.stars, it, release.await())) } } } } Introducing Flows in Gears # A similar abstraction of Kotlin Flows can be implemented in Scala Gears leveraging Tasks and TerminableChannels. The following section describes the attempt made to implement it and what has been achieved. When building the Flow, the client provides a block of code through which emits values, which is wrapped inside a Task that is started only when the collect method is called; The values are sent (emitted) on a TerminableChannel which is created when the collect method is called; the behavior of the emit method is defined inside the apply method of Flow and injected inside caller code via the context parameter (it: FlowCollector[T]) ?=\u0026gt;. Once the task has finished, the channel is terminated. [Source code can be found in commons submodule, pimpimg package.]\n/** An asynchronous cold data stream that emits values, inspired to Kotlin Flows. */ trait Flow[+T]: /** Start the flowing of data which can be collected reacting through the given [[collector]] function. */ def collect(collector: Try[T] =\u0026gt; Unit)(using Async, AsyncOperations): Unit /** An interface modeling an entity capable of [[emit]]ting [[Flow]]s values. */ trait FlowCollector[-T]: /** Emits a value to the flow. */ def emit(value: T)(using Async): Unit object Flow: /** Creates a new asynchronous cold [[Flow]] from the given [[body]]. * Since it is cold, it starts emitting values only when the [[Flow.collect]] method is called. * To emit a value use the [[FlowCollector]] given instance. */ def apply[T](body: (it: FlowCollector[T]) ?=\u0026gt; Unit): Flow[T] = val flow = FlowImpl[T]() flow.task = Task: val channel = flow.channel flow.sync.release() val collector: FlowCollector[T] = new FlowCollector[T]: override def emit(value: T)(using Async): Unit = channel.send(Success(value)) try body(using collector) catch case e: Exception =\u0026gt; channel.send(Failure(e)) flow private class FlowImpl[T] extends Flow[T]: private[Flow] var task: Task[Unit] = uninitialized private[Flow] var channel: TerminableChannel[Try[T]] = uninitialized private[Flow] val sync = Semaphore(0) override def collect(collector: Try[T] =\u0026gt; Unit)(using Async, AsyncOperations): Unit = val myChannel = TerminableChannel.ofUnbounded[Try[T]] synchronized: channel = myChannel task.run.onComplete(() =\u0026gt; myChannel.terminate()) // Ensure to leave the synchronized block after the task has been initialized // with the correct channel instance. sync.acquire() myChannel.foreach(t =\u0026gt; collector(t)) map and flatMap combinators have been implemented on top of Flow:\nobject FlowOps: extension [T](flow: Flow[T]) /** @return a new [[Flow]] whose values has been transformed according to [[f]]. */ def map[R](f: T =\u0026gt; R): Flow[R] = new Flow[R]: override def collect(collector: Try[R] =\u0026gt; Unit)(using Async, AsyncOperations): Unit = catchFailure(collector): flow.collect(item =\u0026gt; collector(Success(f(item.get)))) /** @return a new [[Flow]] whose values are created by flattening the flows generated * by the given function [[f]] applied to each emitted value of this. */ def flatMap[R](f: T =\u0026gt; Flow[R]): Flow[R] = new Flow[R]: override def collect(collector: Try[R] =\u0026gt; Unit)(using Async, AsyncOperations): Unit = catchFailure(collector): flow.collect(item =\u0026gt; f(item.get).collect(x =\u0026gt; collector(Success(x.get)))) Showcasing Flows # Library use case:\ntype Name = String type WriterId = Int type Writer = (Name, WriterId) type Book = String object LibraryService: private val users: Set[Writer] = Set((\u0026#34;Alice\u0026#34;, 987), (\u0026#34;Bob\u0026#34;, 123), (\u0026#34;Charlie\u0026#34;, 342)) private val books: Map[WriterId, Set[Book]] = Map( 987 -\u0026gt; Set(\u0026#34;Alice\u0026#39;s Adventures in Wonderland\u0026#34;, \u0026#34;Through the Looking-Glass\u0026#34;), 123 -\u0026gt; Set(\u0026#34;The Shining\u0026#34;), 342 -\u0026gt; Set(\u0026#34;The Raven\u0026#34;, \u0026#34;The Tell-Tale Heart\u0026#34;), ) def allWriters(using Async): Flow[Writer] = Flow: users.foreach { u =\u0026gt; sleep(2_000) it.emit(u) } def booksByWriter(writer: WriterId)(using Async): Flow[Book] = Flow: books(writer).foreach(it.emit) Flows are cold\n@main def useSimple(): Unit = Async.blocking: val service = LibraryService val writers = service.allWriters log(s\u0026#34;Not collecting yet!\u0026#34;) sleep(1_000) // something meaningful log(\u0026#34;Starting collecting users...\u0026#34;) writers.collect(u =\u0026gt; log(s\u0026#34;User: $u\u0026#34;)) println(\u0026#34;Done\u0026#34;) What we get is something like:\n[1709559932492] Not collecting yet! [1709559933500] Starting collecting users... [1709559935532] User: Success((Alice,987)) [1709559937536] User: Success((Bob,123)) [1709559939541] User: Success((Charlie,342)) Done If something goes wrong during the task execution, a Failure is emitted and the task terminates (no more values are emitted). For example:\ndef failingWriters(using Async): Flow[Writer] = Flow: throw IllegalStateException(\u0026#34;Library is closed\u0026#34;) it.emit(users.head) @main def useFailingFlow(): Unit = Async.blocking: val service = LibraryService val writers = service.failingWriters writers.collect(println) Results in:\nFailure(java.lang.IllegalStateException: The library is closed) Flows can be transformed\nmapping Writers to their identifier:\n@main def useWithMapping(): Unit = Async.blocking: val service = LibraryService val writersId = service.allWriters.map(_._2) writersId.collect(a =\u0026gt; println(s\u0026#34;Id: $a\u0026#34;)) Result:\nId: Success(987) Id: Success(123) Id: Success(342) flatMapping to get all the books:\n@main def useWithFlatMap(): Unit = Async.blocking: val service = LibraryService val allBooks = service.allWriters.flatMap(w =\u0026gt; service.booksByWriter(w._2) ) allBooks.collect(println) Result:\nSuccess(Alice\u0026#39;s Adventures in Wonderland) Success(Through the Looking-Glass) Success(The Shining) Success(The Raven) Success(The Tell-Tale Heart) 👉🏻 More tests on Flows can be found in commons, pimping pakcage.\nTakeaways # Channels are the basic communication and synchronization primitive for exchanging data between Futures/Coroutines. Scala Gears support for Terminable channels or a review of the closing mechanism should be considered. The Flow abstraction in Kotlin Coroutines is a powerful tool for handling cold streams of data, and it is a perfect fit for functions that need to return a stream of asynchronously computed values upon request. A similar abstraction can be implemented in Scala Gears leveraging Tasks and TerminableChannels, enabling improved support for an asynchronous flow of data also in Gears, which is currently lacking. Previous: Basic asynchronous constructs Next: Reactivity in direct style "},{"id":4,"href":"/direct-style-experiments/docs/05-rears/","title":"05 Rears","section":"Docs","content":" Reactivity in direct style # So far, we\u0026rsquo;ve explored the basics of asynchronous abstraction mechanisms provided by the direct style of the Scala Gears and Kotlin Coroutines frameworks. The goal of this last example is to investigate, using a simple example, whether these two frameworks offer sufficient idiomatic abstractions to deal with event-based reactive systems.\nSmart Hub example # Idea: in an IoT context, a multitude of sensors of different types, each replicated to ensure accuracy, transmit their measurements to a central hub, which in turn needs to react, in real-time, forwarding to the appropriate controller the data, possibly performing some kind of transformation. Scala Gears version # Before delving into the example, two abstractions of Gears, yet not covered, are introduced:\nTasks provide a way, not only to run asynchronous computation, essentially wrapping a () =\u0026gt; Future[T], but also to schedule it, possibly repeating it. Different scheduling strategies are available: Every, ExponentialBackoff, FibonacciBackoff, RepeatUntilFailure, RepeatUntilSuccess. This allows for implementing easily proactive computations classDiagram class `Task[+T]` { +apply(body: (Async, AsyncOperations) ?=\u003e T) Task[T]$ +run(using Async, AsyncOperations) Future[+T] +schedule(s: TaskSchedule) Task[T] } `Task[+T]` o--\u003e TaskSchedule class TaskSchedule { \u003c\u003c enum \u003e\u003e + Every(millis: Long, maxRepetitions: Long = 0) + ExponentialBackoff(millis: Long, exponentialBase: Int = 2, maxRepetitions: Long = 0) + FibonacciBackoff(millis: Long, maxRepetitions: Long = 0) + RepeatUntilFailure(millis: Long = 0, maxRepetitions: Long = 0) + RepeatUntilSuccess(millis: Long = 0, maxRepetitions: Long = 0) } Warning: when Tasks are scheduled with RepeatUntil*:\nif the body of a Task does not perform any suspending operations the Async.blocking blocks the current thread until the task is completed (either successfully or not); if the body of a Task does perform suspending operations then the Async.blocking does not wait for the task to complete and its context is left as soon as reaches its end. If we want to wait for the task completion, it\u0026rsquo;s the client\u0026rsquo;s responsibility to explicitly Async.await (or awaitResult) Cons: depending on the content of the block, the behavior is different! This is error-prone! Warning: with high-order functions if we deal with repeated Tasks, in some cases an Async ?=\u0026gt; label is required to not suspend the whole block, even if a suspending operation is performed: the code below behaves differently if the Async ?=\u0026gt; label is present or not. Note: this may be an unintended effect of the library, yet to be investigated (sometimes on Ubuntu it doesn\u0026rsquo;t work, suggesting to be a \u0026ldquo;bug\u0026rdquo;, see here vs. here) In this case despite we suspend to wait for the timer tick, the Async.blocking blocks until the Task is completed.\nAsync.blocking: val timer = Timer(2.seconds) Future(timer.run()) produce { _ =\u0026gt; timer.src.awaitResult // SUSPENDING OPERATION! // ... } def produce[T]( action: SendableChannel[T] =\u0026gt; Try[Unit] )(using Async): ReadableChannel[T] = val channel = UnboundedChannel[T]() Task { action(channel.asSendable) }.schedule(RepeatUntilFailure()).run channel.asReadable With the Async ?=\u0026gt; label, the Async.blocking does not wait for the Task to complete!\nAsync.blocking: val timer = Timer(2.seconds) Future(timer.run()) produceWithLabel { _ =\u0026gt; timer.src.awaitResult // SUSPENDING OPERATION! // .... } def produceWithLabel[T]( action: Async ?=\u0026gt; SendableChannel[T] =\u0026gt; Try[Unit] )(using Async): ReadableChannel[T] = val channel = UnboundedChannel[T]() Task { action(channel.asSendable) }.schedule(RepeatUntilFailure()).run channel.asReadable [See the tests for more details.]\nTo avoid the work-stealing behavior of channel consumers, a ChannelMultiplexer can be used. It is essentially a container of Readable and Sendable channels, which can be added and removed at runtime. Internally, it is implemented with a thread that continuously races the set of publishers and once it reads a value, it forwards it to each subscriber channel. Order is guaranteed only per producer; Typically, the consumer creates a channel and adds it to the multiplexer, then starts reading from it, possibly using a scheduled task. if the consumer attaches to the channel after the producer has started, the values sent during this interval are lost, like hot observables in Rx. classDiagram namespace javaio { class Closeable { \u003c\u003c interface \u003e\u003e +close() } } class `ChannelMultiplexer[T]` { \u003c\u003c trait \u003e\u003e +run()(using Async) +addPublisher(c: ReadableChannel[T]) +removePublisher(c: ReadableChannel[T]) +addSubscriber(c: SendableChannel[Try[T]]) +removeSubscriber(c: SendableChannel[Try[T]]) } Closeable \u003c|-- `ChannelMultiplexer[T]` In the proposed strawman Scala Gears library, there are no other kinds of abstractions, nor a way to manipulate channels with functions inspired by Rx.\nThe attempt, described in the following, has been to extend this framework adding first-class support for Producer and Consumer\u0026rsquo;s concepts and implementing some of the most common Rx operators, completely leaving out performance concerns.\n[Sources can be found in the rears submodule.].\nA Producer is a runnable entity, programmed with a Task, producing items on a channel. It exposes the publishingChannel method, which returns a ReadableChannel through which interested consumers can read produced items. A Consumer is a runnable entity devoted to consuming data from a channel, exposed by the listeningChannel method which returns a SendableChannel to send items to. It can be made stateful by mixing it with the State trait, allowing it to keep track of its state, which is updated every time with the result of the reaction (i.e. its return type). Warning Like in an event-loop, the reaction logic should not perform long-lasting blocking operation, otherwise, the whole system will not react to new events: the Async capability is though needed if you want to give the client the ability to invoke Futures within this block; otherwise, another option (alternative to the following) would be to encapsulate the reaction behavior within a Task and run it at every received data. However, race conditions could take place in this last case. /** A producer, i.e. a runnable entity producing items on a channel. */ trait Producer[E]: /** The [[Channel]] where specific [[Producer]]s send items to. */ protected val channel: Channel[E] = UnboundedChannel() /** @return the publisher\u0026#39;s behavior encoded as a runnable [[Task]]. */ def asRunnable: Task[Unit] /** @return the [[ReadableChannel]] where produced items are placed. */ def publishingChannel: ReadableChannel[E] = channel.asReadable /** A consumer, i.e. a runnable entity devoted to consume data from a channel. */ trait Consumer[E, S]: /** The [[SendableChannel]] to send items to, where the consumer listen for new items. */ val listeningChannel: SendableChannel[Try[E]] = UnboundedChannel() /** @return a runnable [[Task]]. */ def asRunnable: Task[Unit] = Task { listeningChannel.asInstanceOf[Channel[Try[E]]].read().foreach(react) }.schedule(RepeatUntilFailure()) /** The suspendable reaction triggered upon a new read of an item succeeds. */ protected def react(e: Try[E])(using Async): S /** A mixin to make consumer stateful. Its state is updated with the result * of the [[react]]ion. Initially its state is set to [[initialValue]]. */ trait State[E, S](initialValue: S): consumer: Consumer[E, S] =\u0026gt; private var _state: S = initialValue /** @return the current state of the consumer. */ def state: S = synchronized(_state) override def asRunnable: Task[Unit] = Task { listeningChannel.asInstanceOf[Channel[Try[E]]].read().foreach { e =\u0026gt; synchronized { _state = react(e) } } }.schedule(RepeatUntilFailure()) The Controller object exposes methods wiring Producer and Consumers altogether, possibly performing some kind of transformation on the publisherChannel. the oneToOne method just wires one single consumer to the publisherChannel given in input, possibly having it transformed with the provided transformation. the oneToMany allows many consumers to be wired to the publisherChannel, possibly having it transformed. to accomplish this, a ChannelMultiplexer is used, which is in charge of forwarding the items read from the transformed publisherChannel to all consumers\u0026rsquo; channels. /** Simply, a function that, given in input a [[ReadableChannel]], performs some * kind of transformation, returning, as a result, another [[ReadableChannel]]. */ type PipelineTransformation[T, R] = ReadableChannel[T] =\u0026gt; ReadableChannel[R] object Controller: /** Creates a runnable [[Task]] forwarding the items read from the [[publisherChannel]] * to the given [[consumer]], after having it transformed with the given [[transformation]]. */ def oneToOne[T, R]( publisherChannel: ReadableChannel[T], consumer: Consumer[R, ?], transformation: PipelineTransformation[T, R] = identity, ): Task[Unit] = val transformedChannel = transformation(publisherChannel) Task { consumer.listeningChannel.send(transformedChannel.read().tryable) }.schedule(RepeatUntilFailure()) /** Creates a runnable [[Task]] forwarding the items read from the [[publisherChannel]] to * all consumers\u0026#39; channels, after having it transformed with the given [[transformation]]. */ def oneToMany[T, R]( publisherChannel: ReadableChannel[T], consumers: Set[Consumer[R, ?]], transformation: PipelineTransformation[T, R] = identity, ): Task[Unit] = Task: val multiplexer = ChannelMultiplexer[R]() consumers.foreach(c =\u0026gt; multiplexer.addSubscriber(c.listeningChannel)) multiplexer.addPublisher(transformation(publisherChannel)) // blocking call: the virtual thread on top of which this task is // executed needs to block to continue publishing publisher\u0026#39;s events // towards the consumer by means of the multiplexer. multiplexer.run() The following PipelineTransformations have been implemented (inspired by Rx). Tests in rears submodule provide the necessary examples to understand their behavior.\nFilter # /** @return a new [[ReadableChannel]] whose elements passes the given predicate [[p]]. */ def filter(p: T =\u0026gt; Boolean): ReadableChannel[T] Example:\n----1---2-------3----4---5--6----7---8---9---10---\u0026gt; | | | | | | | | | | ----V---V-------V----V---V--V----V---V---V---V----- filter(_ % 2 == 0) --------|--------------|------|-------|---------|-- V V V V V --------2--------------4------6-------8--------10-\u0026gt; Map # /** @return a new [[ReadableChannel]] whose values are transformed accordingly to the given function [[f]]. */ def map[R](f: T =\u0026gt; R): ReadableChannel[R] Example:\n----1---2-------3----4---5------6--------7--------\u0026gt; | | | | | | | ----V---V-------V----V---V------V--------V--------- map(x =\u0026gt; x * x) ----|---|-------|----|---|------|--------|--------- V V V V V V V ----1---4-------9----16--25-----36-------49-------\u0026gt; Debounce # /** @return a new [[ReadableChannel]] whose elements are emitted only after * the given [[timespan]] has elapsed since the last emission. */ def debounce(timespan: Duration): ReadableChannel[T] Example:\n----1---2-------3----4---5--6-----7---8---9---10--\u0026gt; | | | | | | | | | | V V V V V V V V V V T----------T----------T----------T----------T------ debounce(1 second) --------------------------------------------------- | | | | | V V V V V -------1---------3---------5------7------------10-\u0026gt; GroupBy # /** Groups the items emitted by a [[ReadableChannel]] according to the given [[keySelector]]. * @return key-value pairs, where the keys are the set of results obtained from applying the * [[keySelector]] coupled to a new [[ReadableChannel]] where only items belonging to * that grouping are emitted. */ def groupBy[K](keySelector: T =\u0026gt; K): ReadableChannel[(K, ReadableChannel[T])] Example:\n----1---2-3--4---5--6---\u0026gt; | | | | | | V V V V V V ------------------------- groupBy(_ % 2) ------------------------- \\ \\ ----false--true------------\u0026gt; 1 2 \\ \\ \\ 4 3 \\ \\ \\ 5 6 Buffer # /** @return a new [[ReadableChannel]] whose elements are buffered in a [[List]] of size [[n]]. * If [[timespan]] duration is elapsed since last read the list is emitted * with collected elements until that moment (default: 5 seconds). */ def buffer(n: Int, timespan: Duration = 5 seconds): ReadableChannel[List[T]] Example:\n----1---2-3----4---5--6----7---8--------\u0026gt; | | | | | | | | V V V V V V V V |---------|-----------|------------T----- buffer(n = 3, timespan = 5 seconds) |---------|-----------|------------|----- V V V ------[1, 2, 3]---[4, 5, 6]------[7, 8]-\u0026gt; BufferWithin # /** @return a new [[ReadableChannel]] whose elements are buffered in a [[List]] of items * if emitted within [[timespan]] duration after the first one (default: 5 seconds). */ def bufferWithin(timespan: Duration = 5 seconds): ReadableChannel[List[T]] Example:\n----1---2-3-4---5--6--7----------8-----------\u0026gt; | | | | | | | | V V V V V V V V ----|--------T--|--------T-------|--------T--- buffer(timespan = 5 seconds) -------------|-----------|----------------|--- V V V -------[1, 2, 3, 4]--[5, 6, 7]-----------[8]-\u0026gt; Going back to the example here is presented a schema summarizing the flows of data and the transformations to apply to them. This is just a simple example used to test the proposed abstractions.\n[Sources are available in smart-hub-direct submodule].\nFor simplicity, two types of sensors are considered: TemperatureSensor and LuminositySensor;\nsensors send data to the smart hub SensorSource (e.g., in a real case scenario, via MQTT)\nSensorSource is a Producer[SensorEvent], publishing received data on its publishingChannel:\ntrait SensorSource extends Producer[SensorEvent] sealed trait SensorEvent(val name: String) case class TemperatureEntry(sensorName: String, temperature: Temperature) extends SensorEvent(sensorName) case class LuminosityEntry(sensorName: String, luminosity: Temperature) extends SensorEvent(sensorName) three main controllers:\nSensorHealthChecker is a stateful consumer of generic SensorEvents that checks the health of the sensors, sending alerts in case of malfunctioning. Here the state is necessary to determine the health of the sensors, based on the last detection.\n/** A [[state]]ful consumer of [[SensorEvent]] detecting possible * malfunctioning and keeping track of last known active sensing units. */ trait SensorHealthChecker extends Consumer[Seq[E], Seq[E]] with State[Seq[E], Seq[E]] The Thermostat is a stateful consumer of temperature entries, taking care of controlling the heating system. The fact the thermostat keeps track of the last average detection could be useful to a ReSTful API, for example.\n/** A [[state]]ful consumer of [[TemperatureEntry]]s in charge of controlling * the heater and keeping track of the last detected average temperature. */ trait Thermostat extends Consumer[Seq[TemperatureEntry], Option[Temperature]] with State[Seq[TemperatureEntry], Option[Temperature]]: val scheduler: T LightingSystem is a basic consumer (non-stateful) of luminosity entries, taking care of controlling the lighting system.\n/** A consumer of [[LuminosityEntry]], in charge of controlling the lighting system. */ trait LightingSystem extends Consumer[Seq[LuminosityEntry], Unit] Each of these controllers reacts to the data received based on their logic and their actual state to accomplish a specific task. For example:\nthe sensor checker sends alerts whether, compared with the previous detection, it did not receive data from some sensors:\noverride protected def react(e: Try[Seq[E]])(using Async): Seq[E] = e match case Success(current) =\u0026gt; val noMoreActive = state.map(_.name).toSet -- current.map(_.name).toSet if noMoreActive.nonEmpty then sendAlert(s\u0026#34;[${currentTime}] Detected ${noMoreActive.mkString(\u0026#34;, \u0026#34;)} no more active!\u0026#34;) current case Failure(es) =\u0026gt; sendAlert(es.getMessage); Seq() the thermostat computes the average temperature and, based on a scheduler, decides whether to turn on or off the heating system:\noverride protected def react(e: Try[Seq[TemperatureEntry]])(using Async): Option[Temperature] = for averageTemperature \u0026lt;- e.map { entries =\u0026gt; entries.map(_.temperature).sum / entries.size }.toOption _ = averageTemperature.evaluate() // here logic to decide whether turn on or off heating system yield averageTemperature The HubManager takes care of grouping sensor data by their type and forwarding them to the appropriate manager, either ThermostatManager or LightingManager:\nval channelBySensor = sensorsSource.publishingChannel.groupBy(_.getClass) Task { channelBySensor.read() match case Right((clazz, c)) if clazz == classOf[TemperatureEntry] =\u0026gt; thermostatManager.run(c.asInstanceOf[ReadableChannel[TemperatureEntry]]) case Right((clazz, c)) if clazz == classOf[LuminosityEntry] =\u0026gt; lightingManager.run(c.asInstanceOf[ReadableChannel[LuminosityEntry]]) case _ =\u0026gt; () }.schedule(RepeatUntilFailure()).run sensorsSource.asRunnable.run.await Both ThermostatManager and LightingManager are in charge of creating the appropriate Controller instance, based on the number of Consumers and pipeline transformation we need to implement:\n// ThermostatManager def run(source: ReadableChannel[TemperatureEntry])(using Async, AsyncOperations): Unit = thermostat.asRunnable.run sensorHealthChecker.asRunnable.run Controller.oneToMany( publisherChannel = source, consumers = Set(thermostat, sensorHealthChecker), transformation = r =\u0026gt; r.bufferWithin(samplingWindow), ).run To produce a testable version of this example, a simulated source of sensor data has been created, backed to a GUI, through which the user can simulate the behavior of the sensors. The example is runnable via:\n./gradlew smart-hub-\u0026lt;direct | direct-kt\u0026gt;:run Three panels should pop up, one for each sensor type, and a dashboard showing the state of the system. Entering some value in the panels and pressing the \u0026ldquo;Send\u0026rdquo; button, after 10 seconds (the configured sampling window), the system should react to the data received, updating the dashboard with the new state.\nKotlin Coroutines version # Kotlin Coroutines offers two other abstractions to deal with asynchronous data streams, belonging to the flow \u0026ldquo;family\u0026rdquo;, which are: SharedFlow and StateFlow. Despite their names including flow, which we\u0026rsquo;ve seen are cold streams, they are actually hot (the terminology is a bit misleading\u0026hellip;):\nSharedFlow is a hot flow that allows for multiple collectors to subscribe to it, enabling the broadcasting of values to multiple consumers or having multiple consumers be \u0026ldquo;attached\u0026rdquo; to the same stream of data. they can be configured to buffer a certain number of previously emitted values for new collectors so that they can catch up with the latest values \u0026ndash; the so-called, replay cache; StateFlow is an extension of the SharedFlow: it is a hot flow that maintains a single value representing a state, holding one value at a time. It operates as a conflated flow, meaning that when a new value is emitted, it replaces the previous value and is immediately sent to new collectors this type of flow is beneficial for maintaining a single source of truth for a state and automatically updating all collectors with the latest state (for example in ViewModels in Android applications) In our example, SharedFlow is used to model the flow of sensor data:\ninterface SensorSource\u0026lt;out E : SensorEvent\u0026gt; { /** The flow of sensor events. */ val sensorEvents: SharedFlow\u0026lt;E\u0026gt; } Like all flows, they have all the kinds of operators presented in the previous example. Despite this, they do not support, at the moment, all the operators that Rx offers, like groupBy, buffer (in the Rx conception), etc\u0026hellip; (even if some proposals are pending to add them in the framework).\nFor this reason, the consumer of events has been implemented manually, using a loop that, every samplingWindow time, reacts to the data received, updating the state of the system. By the way, if this solution appears to be less elegant, since Flows are, de facto, the porting of Rx\u0026rsquo;s Observable's into the Coroutines world, libraries exists to convert them to Observable and vice versa. This could offer (in some cases and where necessary) a way to use the full power of Rx operator, if needed.\n/** A consumer of sensor events. */ interface SensorSourceConsumer\u0026lt;in E : SensorEvent, out S\u0026gt; { /** The current state of the source consumer. */ val state: S /** Reacts to a sensor event. */ suspend fun react(e: E) } /** A scheduled consumer of sensor events. */ interface ScheduledConsumer\u0026lt;in E : SensorEvent, out S\u0026gt; : SensorSourceConsumer\u0026lt;E, S\u0026gt;, CoroutineScope { /** The interval period. */ val samplingWindow: Duration /** The update logic of the consumer. */ suspend fun update() /** Runs the consumer scheduler. */ fun run() = launch { while (true) { update() delay(samplingWindow) } } } The managers just take care of collecting the data and forwarding it to the appropriate consumer. For example, the ThermostatManager:\nsuspend fun run(sensorSource: Flow\u0026lt;TemperatureEntry\u0026gt;) { thermostat.run() temperatureSensorsChecker.run() sensorSource.collect { thermostat.react(it) temperatureSensorsChecker.react(it) } } Takeaways # Channels in Scala Gears are good to model flow of data that exist without application\u0026rsquo;s request from them: incoming network connections, event streams, etc\u0026hellip; The scheduling mechanism of Task, along with the mutiplexer abstraction, despite having some stability issues, allows to implement flows of hot data which are listened by multiple consumers. Transformation operators inspired by the Reactive world could enhance the expressiveness of the framework, making it more suitable for modeling reactive event-based systems. Previous: Channels as a communication primitive Home "}]
\ No newline at end of file
diff --git a/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js b/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js
similarity index 90%
rename from en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js
rename to en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js
index e40f73c2..22f51acc 100644
--- a/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js
+++ b/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js
@@ -1 +1 @@
-"use strict";(function(){const o="/direct-style-experiments/en.search-data.min.a12762fe0d0ac0594b436c422d039d4c34cf9d6bf009a2124993806d3e50e428.json",i=Object.assign({cache:!0},{doc:{id:"id",field:["title","content"],store:["title","href","section"]}}),e=document.querySelector("#book-search-input"),t=document.querySelector("#book-search-results");if(!e)return;e.addEventListener("focus",n),e.addEventListener("keyup",s),document.addEventListener("keypress",a);function a(t){if(t.target.value!==void 0)return;if(e===document.activeElement)return;const n=String.fromCharCode(t.charCode);if(!r(n))return;e.focus(),t.preventDefault()}function r(t){const n=e.getAttribute("data-hotkeys")||"";return n.indexOf(t)>=0}function n(){e.removeEventListener("focus",n),e.required=!0,fetch(o).then(e=>e.json()).then(e=>{window.bookSearchIndex=FlexSearch.create("balance",i),window.bookSearchIndex.add(e)}).then(()=>e.required=!1).then(s)}function s(){for(;t.firstChild;)t.removeChild(t.firstChild);if(!e.value)return;const n=window.bookSearchIndex.search(e.value,10);n.forEach(function(e){const n=c("<li><a href></a><small></small></li>"),s=n.querySelector("a"),o=n.querySelector("small");s.href=e.href,s.textContent=e.title,o.textContent=e.section,t.appendChild(n)})}function c(e){const t=document.createElement("div");return t.innerHTML=e,t.firstChild}})()
\ No newline at end of file
+"use strict";(function(){const o="/direct-style-experiments/en.search-data.min.5cd36aacfc646802f6043b59de31ec5fe9b0983544fd189d576e27dcc8106546.json",i=Object.assign({cache:!0},{doc:{id:"id",field:["title","content"],store:["title","href","section"]}}),e=document.querySelector("#book-search-input"),t=document.querySelector("#book-search-results");if(!e)return;e.addEventListener("focus",n),e.addEventListener("keyup",s),document.addEventListener("keypress",a);function a(t){if(t.target.value!==void 0)return;if(e===document.activeElement)return;const n=String.fromCharCode(t.charCode);if(!r(n))return;e.focus(),t.preventDefault()}function r(t){const n=e.getAttribute("data-hotkeys")||"";return n.indexOf(t)>=0}function n(){e.removeEventListener("focus",n),e.required=!0,fetch(o).then(e=>e.json()).then(e=>{window.bookSearchIndex=FlexSearch.create("balance",i),window.bookSearchIndex.add(e)}).then(()=>e.required=!1).then(s)}function s(){for(;t.firstChild;)t.removeChild(t.firstChild);if(!e.value)return;const n=window.bookSearchIndex.search(e.value,10);n.forEach(function(e){const n=c("<li><a href></a><small></small></li>"),s=n.querySelector("a"),o=n.querySelector("small");s.href=e.href,s.textContent=e.title,o.textContent=e.section,t.appendChild(n)})}function c(e){const t=document.createElement("div");return t.innerHTML=e,t.firstChild}})()
\ No newline at end of file
diff --git a/index.html b/index.html
index 2bcd2e1d..7462a48f 100644
--- a/index.html
+++ b/index.html
@@ -20,7 +20,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <link rel="alternate" type="application/rss+xml" href="https://tassiluca.github.io/direct-style-experiments/index.xml" title="direct-style-experiments" />
 <!--
 Made with Book Theme
diff --git a/sitemap.xml b/sitemap.xml
index 9a6c3382..3f24e4d4 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -6,7 +6,7 @@
     <lastmod>2024-03-07T12:58:54+01:00</lastmod>
   </url><url>
     <loc>https://tassiluca.github.io/direct-style-experiments/docs/01-overview/</loc>
-    <lastmod>2024-03-07T12:58:54+01:00</lastmod>
+    <lastmod>2024-03-07T13:31:46+01:00</lastmod>
   </url><url>
     <loc>https://tassiluca.github.io/direct-style-experiments/docs/02-boundaries/</loc>
     <lastmod>2024-03-07T12:58:54+01:00</lastmod>
@@ -23,7 +23,7 @@
     <loc>https://tassiluca.github.io/direct-style-experiments/categories/</loc>
   </url><url>
     <loc>https://tassiluca.github.io/direct-style-experiments/docs/</loc>
-    <lastmod>2024-03-07T12:58:54+01:00</lastmod>
+    <lastmod>2024-03-07T13:31:46+01:00</lastmod>
   </url><url>
     <loc>https://tassiluca.github.io/direct-style-experiments/tags/</loc>
   </url>
diff --git a/tags/index.html b/tags/index.html
index 316c3525..bb1f503d 100644
--- a/tags/index.html
+++ b/tags/index.html
@@ -17,7 +17,7 @@
 <link rel="canonical" href="https://tassiluca.github.io/direct-style-experiments/tags/">
 <link rel="stylesheet" href="/direct-style-experiments/book.min.42504dd4648e8da78376617fd32b73c3ae6afbe9f805c36868bc831b07002332.css" integrity="sha256-QlBN1GSOjaeDdmF/0ytzw65q&#43;&#43;n4BcNoaLyDGwcAIzI=" crossorigin="anonymous">
   <script defer src="/direct-style-experiments/flexsearch.min.js"></script>
-  <script defer src="/direct-style-experiments/en.search.min.3f8b6945cd08ef775004f85d77654ec92ace96dfa5464784bf41f620c35ae240.js" integrity="sha256-P4tpRc0I73dQBPhdd2VOySrOlt&#43;lRkeEv0H2IMNa4kA=" crossorigin="anonymous"></script>
+  <script defer src="/direct-style-experiments/en.search.min.bf3e6c4047a74032e851536a446a84e22354d1e9cd8c7448337ec5431e8104e4.js" integrity="sha256-vz5sQEenQDLoUVNqRGqE4iNU0enNjHRIM37FQx6BBOQ=" crossorigin="anonymous"></script>
 <link rel="alternate" type="application/rss+xml" href="https://tassiluca.github.io/direct-style-experiments/tags/index.xml" title="direct-style-experiments" />
 <!--
 Made with Book Theme