Metaphor</strong></th>	Type state</strong></th>	Ownership event</strong></th></tr></thead>
undressed</em></td>	`Type</code></td>`	moveable / free</td></tr>
dress-up</em></td>	`Pin::new(Type)</code></td>`	give up ownership</td></tr>
dressed-up</em></td>	`Pin<Type></code></td>`	stuck in memory</td></tr>
dress-down</em></td>	`Pin::new(Type).get_mut()</code></td>`	acquire edit access</td></tr> </tbody></table> Conclusion:</p> `Pin</code> does not pin anything physically at run-time, but relies on the auto-trait !Unpin</code> to prevent dangerous moves at run-time.</p> </blockquote>` Convert !Unpin</code> into Unpin</code></a></h3> Almost all primitives in Rust are Unpin</code>. Whenever you encounter something that is not Unpin</code> (and has not non-'static</code> references), you can just allocate it on the heap with the function Box::pin</code>. The Box::pin</code> is shorthand for Pin::new(Box::new())</code>.</p> The result of the Box::pin</code> is a combination:</p> The heap-allocated pinned object will not be moved by any code generated by the Rust compiler;</li> We can still drop the pinned object and the Drop</code> implementations will run.</li> </ul> Let’s take the following as an example of something that is !Unpin</code>.</p> struct MapFut<Fut> where Fut: Future { fut: Fut, ... } </code></pre> A common approach when handling extensions or combinators of futures which are themselves futures, is by projecting</strong> Pin<&mut Self></code>into &mut Self</code>. The projection is usually called this</code> (the self</code> keyword is reserved).</p> impl<Fut> Future for MapFut<Fut> { fn poll(self: Pin<&mut Self>) -> Poll { // We need to do something with `self.fut`. // We need `Pin<&mut Self> -> &mut Self`. // The following will not compile, because `Self: !Unpin`. let mut this = self.get_mut(); unimplemented!() } } </code></pre> In this example the user struct MapFut</code> is !Unpin</code>, so get_mut</code> cannot be used. The get_mut</code> function requires Self: Unpin</code> as mentioned in the previous section.</p> The simplest solution is to refine the definition of MapFut</code> as follows:</p> struct MapFut<Fut> where Fut: Future { fut: Pin<Box<Fut>>, ... } </code></pre> Now fut</code> is a pinned pointer to a location on the heap.</p> Unsafe Pin</code> methods (optional)</a></h3> Until now I explicitly avoided mentioning or using the unsafe methods of the Pin</code> type. Beside Pin::new()</code> and Pin::get_mut()</code>, Pin</code> also has a few unsafe</code> counterparts.</p> You might have been tempted by the Rust compiler to use the unsafe</code> methods because they do not have an Unpin</code> constraint. But if you go in that direction, you effectively disable automatic and important checks by the compiler at run-time.</p> The Ready</code> future</a></h3> Anything can be turned into a future using the Ready</code> future. This future takes the source and puts it inside an Option</code>.</p> pub struct Ready<T>(Option<T>); </code></pre> This future never has to wake up, because it yields a value immediately and is read on the first poll</code> call. This is why the context (and its waker) are ignored.</p> impl<T> Future for Ready<T> { type Output = T; fn poll(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<T> { Poll::Ready(self.0.take().expect("Ready polled after completion")) } } </code></pre> To use this future, you can directly call the constructor Ready::new()</code> or you can call ready()</code>:</p> assert_eq!(1, ready(1).await)`; </code></pre> Remark: .await</code> automatically converts everything in a future with IntoFuture</code>, so it might not even be necessary to use Ready</code> explicitly.</em></p> Simple stream combinators</a></h2> Definition of a stream</a></h3> A stream is an future that may be polled more than once and yield more than one Poll::Ready</code> value.</p> The official definition is a trait:</p> pub trait Stream { type Item; fn poll_next( self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll<Option<Self::Item>>; } </code></pre> Notice the similary with the Future</code> trait.</p> Important: As of April 2025, the Stream</code> trait is not yet in stable Rust.</em></p> Futures as streams with Once</code></a></h3> Every future Fut</code> can be converted into a stream by returning Some(Fut::Output)</code>:</p> pub struct Once<Fut> { future: Option<Fut> } impl<Fut: Future> Stream for Once<Fut> { type Item = Fut::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { let mut this = self.project(); let v = match this.future.as_mut().as_pin_mut() { Some(fut) => ready!(fut.poll(cx)), None => return Poll::Ready(None), }; this.future.set(None); Poll::Ready(Some(v)) } } </code></pre> Here, ready!</code> converts Poll</code> into Option</code>.</p> Functional building block Map</code></a></h3> Let’s look at a simple example from the “semi-standard” library crate futures</code>: the map</code> combinator. This combinator just maps the items of an input stream and returns an new output stream (while consuming the input stream.)</p> The combinator’s source code looks like this:</p> pub struct Map<St, F> { #[pin] stream: St, f: F, } impl<St, F> Stream for Map<St, F> where St: Stream, F: FnMut<St::Item>, { type Item = F::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { ... } } </code></pre> Ignore the field attribute pin</code> for now.</p> The definition of map</code></a> and all the behaviour that we expect from a map</code> function essentially boils down to storing two things:</p> The original input stream.</li> The function or closure that maps elements of the input stream.</li> </ul> Then the authors of futures</code> had to implement the single required method of the Stream</code> trait: poll_next</code>:</p> impl<St, F> Stream for Map<St, F> where St: Stream, F: FnMut<St::Item>, { type Item = F::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { let mut this = self.project(); let res = ready!(this.stream.as_mut().poll_next(cx)); Poll::Ready(res.map(\|x\| this.f.call_mut(x))) } } </code></pre> At the beginning of the poll_next</code> function, only a Pin<&mut Self></code> is given. It has to be “undressed” to a mutable reference to the state object Map</code>. This happens through a process called projection</strong>, which is essentially just calling get_mut</code> on Pin</code>.</p> Projecting with pin_project</code> (optional)</a></h3> Almost any library I encountered uses the pin_project</code> crate which provides the #[pin]</code> field attribute that I asked you to ignore. The pin_project</code> crate takes care of two things:</p> You don’t have to call Box::pin</code> yourself or pin_mut!</code> inside the constructor function of your aggregated stream.</li> You get a project</code> function which allows you to call as_mut()</code> (equivalent to get_mut()</code>).</li> </ul> Remark</strong>: Internally (if I read the code correctly), pin_project</code> makes use of several unsafe</code> function calls of the Pin</code> type. This is not that strange, considering that most of the standard library is written in unsafe</code>. On the other hand, using Box::pin</code> will not have a significant performance impact and it does not require a macro from an external crate.</p> Stream combinators</a></h2> I will now focus on more complicated stream combinators. These are functions that take one or more input streams and output an output stream. I will call such higher-order functions stream combinators</strong> from now on.</p> Remark: The futures-rx</code></a> crate is a good crate to look for if you want to save time and just use combinators made by someone else.</em></p> Flattening nested streams</a></h3> The futures</code> crate has several types of flatten functions for nested streams. Nested streams are Rust data types S</code> with a trait bound S: Stream<Item: Stream></code>.</p> You could separate flatten combinators in two categories: sequential or concurrent.</p> Sequential flatten combinators will never yield values from multiple inner streams at the same time.</p> flatten</code>: flattens the outer stream by pasting the output from the inner streams consecutively. This is usually not what you want, since most streams are infinite and you don’t want to block the outer stream.</li> A “forgetful” flatten (also called switch</code> in RxJs): a kind of flatten of nested streams that only polls the most recent stream that arrived on the outer stream. This is implemented by switch_map</code></a> in futures-rx</code>.</li> </ul> Concurrent flatten combinators will combine values from multiple inner streams:</p> Concurrent flatten_unordered(None)</code>: flattens by merging as many inner streams as possible, as they arrive on the outer stream. This might seems like a useful function, but it often not what you want.</li> Buffered concurrent flatten_unordered(Some(N))</code>: flattens up-to N different inner streams.</li> </ul> Keep in mind that you can also use the select_all</code> function from futures::stream</code> to flatten in the situation where your outer stream is not a real stream but a finite iterable of streams.</p> A stream of Arc</code>s</a></h3> Sometimes you might want to use the items yielded by a single stream in several places simultaneously. You could wrap all the items yielded by the input stream inside a shared reference Arc</code>. The disadvantage of this approach is that you need (a limited form of) garbage cleaning by reference counting.</p> The futures-rx</code> crate exposes a share</code> method that can be used for splitting an input stream.</p> Looking at the docs</a>, there is an example:</p> use futures::{stream::{StreamExt, self}, future::join}; use futures_rx::{Notification, RxExt}; let stream = stream::iter(0..=3); let stream = stream.share(); let sub_stream_a = stream.clone().map(\|event\| event); let sub_stream_b = stream.clone().map(\|event\| event); assert_eq!((vec![0, 1, 2, 3], vec![0, 1, 2, 3]), join(sub_stream_a.collect::<Vec<_>>(), sub_stream_b.collect::<Vec<_>>()).await); </code></pre> The event</code>s yielded by the shared stream seem to be wrappers around Arc<usize></code>. Since usize</code> is Copy</code>, *event</code> is treated as &usize</code> and the referenced usize</code> automically cloned.</p> Remark: Maybe I am wrong?</em></p> A stream of clones</a></h3> The share</code> operator on streams from futures-rx</code> seems to put all values on the input stream inside a reference-counted Arc</code>.</p> I wanted to derive several output streams from the input stream that yielded clones directly without reference counting.</p> Remark: Maybe I could just have used share</code> and then cloned every element?</em></p> I decided to implement a separate stream cloning trait in a crate clone-stream</code></a>. The high-level API of this crate is quite simple. It is just a call to fork</code> and then stream.clone()</code>:</p> use clone_stream::ForkStream; use futures::{FutureExt, StreamExt, stream}; let non_clone_stream = stream::iter(0..10); let clone_stream = non_clone_stream.fork(); let mut cloned_stream = clone_stream.clone(); </code></pre> Remark: There exist a few other crates that are similar.</em></p> For more information about the clone-stream</code> crate, see my presentation at EuroRust 2025</a>.</p> Role of coroutines 2025-04-01T00:00:00+00:00 In some other posts on this site, you will find ways to create streams from scratch and how to combine them. This post will be about the relationship between the concept of a Stream</code> (or asynchronous iterator) and the other, more familiar, functions present in most programming languages.</p> Most of this post was inspired by a post by without.boats</a>.</p> Simple coroutines</a></h2> Concept of a coroutine</a></h3> Normal functions return output (immediately). They do it only once (or never).</p> A coroutine</strong> is a special kind functions that can:</p> be suspended multiple times</li> be resumed multiple times</li> return once</li> </ul> More specifically, at runtime, coroutines go through a process (with specific terminology):</p> When a coroutine suspends, it yield</strong>s a value to the caller. This is a kind of intermediate return value.</li> After observing (or ignoring) the yielded value, the caller can safely forget about the suspended coroutine (temporarily) and continue with other functions.</li> Later the caller can return to this suspended coroutine. The caller needs to resume the suspended coroutine to wake it up. This step is called resumption</strong>. For resumption some resumption data may need to be provided.</li> </ol> These steps may repeat forever or until the coroutine ends by returning. Returning is distinct from yielding, since it is final. The return value is the last value that can be observed by the caller.</p> Remark</strong>: Coroutines are used internally by the Rust compiler while compiling asynchronous code. The compiler implements a form of “stack-less” co-routines for async {}</code> code-blocks. These blocks are compiled implicitly into coroutines that yield at every await</code>-point.</em></p> Directly constructing coroutines</a></h3> The Coroutine</code> trait definition is an extension of the Future</code> trait:</p> pub trait Coroutine<Resume = ()> { type Yield; type Return; fn resume( self: Pin<&mut Self>, resumption: Resume, ) -> CoroutineState<Self::Yield, Self::Return>; } </code></pre> Notice that we need Pin</code>, similarly to Future</code>. This is because coroutines may be self-referential. The resume</code> function should only be called on coroutines that may move (are Unpin</code>). The reason is probably that they should extend the behaviour of Future</code>s which do require Pin</code>.</p> Important</strong>: The Coroutine</code> trait in Rust is unstable and only available on nightly as of April 2025.</em></p> If you look carefully at the Coroutine</code> trait you could see that (in pseudo-code):</p> type Future<Output> = Coroutine< Resume = Context, Yield = (), Return = Output >; </code></pre> More precisely, a future is a coroutine that yields nothing when suspended. A future needs a Context</code> (containing a Waker</code>) to be resumed or woken.</p> Remark</strong>: The resumption data is provided by a asynchronous run-time to schedule resume</code>s in an efficient way.</em></p> Example of a coroutine</a></h3> The Rust docs contain an example of a coroutine. The coroutine does not need any resumption data, but it yields a number and returns a string on completion:</p> let mut coroutine = #[coroutine] \|\| { yield 1; "foo" }; </code></pre> To use this coroutine</code>, we have to provide an initial chunk of resumption data. By default this is the empty tuple ()</code>. The resumption data is passed to the resume</code> function and used to anticipate the first yield. The first (and last) yield is yield 1</code>.</p> match Pin::new(&mut coroutine).resume(()) { CoroutineState::Yielded(1) => {} _ => panic!("unexpected return from resume"), } </code></pre> The next time resume</code> is called, no yield is encountered and the final return value is returned (a string).</p> match Pin::new(&mut coroutine).resume(()) { CoroutineState::Returned("foo") => {} _ => panic!("unexpected return from resume"), } </code></pre> If our coroutine was a Future</code>, then resume</code> would expect a Context</code> with a Waker</code>.</p> Classification of coroutines</a></h2> Reflecting on the concepts of an iterator, future and stream, we can say that:</p> An iterator</strong> is coroutine that yields an Option</code>.</li> A future</strong> is a coroutine that resumes with a Waker</code>.</li> A stream</strong> is an iterator that resumes with a Waker</code> and yields an Option</code>.</li> </ul> Coroutines are a generalisation of these cases, which can be laid-out in a table:</p> </th> YIELDS</em></th> RESUMES</em></th> RETURNS</em></th></tr></thead> Iterator</code></td> Option</code></td> !</code></td> !</code></td></tr> Future</code>, AsyncFn</code></td> ()</code></td> Waker</code></td> Any</code></td></tr> Stream</code></td> Option</code></td> Waker</code></td> !</code></td></tr> Coroutine</code></td> Any</code></td> Any</code></td> Any</code></td></tr> </tbody></table> In this table, the !</code> symbol stands for never</code>, the type that does not have any runtime value. In other words, never</code> is not constructible. It is used often as the return time of non-terminating functions like infinite loops.</p> For a practical introduction to coroutines in Rust, I recommend Asynchronous Programming in Rust</a>.</p> Functional async 2025-03-18T00:00:00+00:00 Introduction</a></h2> Let’s get started with some definitions you may have heard about already.</p> Declarative programming</strong> is when you write code and make sure that the inputs and outputs of sub-modules behave predictably according to fixed invariants. Logic or constraint solver languages like Prolog or ManyWorlds</a> (written by my friend Jo Devriendt</a>) are part of this family.</p> Functional programming</strong> is building the majority of your code from well-defined and side-effect-free functions. It is a sub-set of declarative programming. Languages like ML, Haskell, OCaml, Lisp are part of this family. They are all extensions of Lambda Calculus.</p> Asynchronous programming</strong> is a kind of programming where you not only have functions that evaluate immediately, but there are also functions that may evaluate in the future</em>. Such functions are called asynchronous functions. JavaScript, C#, and Rust are examples.</p> In the following, I will show how declarative, functional, and asynchronous programming can be combined in the Rust programming language with the semi-standard library crate futures</code>.</p> Use-cases</a></h2> Normal imperative asynchronous programming contains a combination of the async</code>, await</code> keywords with imperative keywords like while</code>, loop</code>, or continue</code>.</p> To illustrate where functional asynchronous programming would be useful, I will give a few examples.</p> Channels</a></h3> One approach that you might take when dealing with different sub-modules in a large project is using channels to communicate between the different modules.</p> This could look like this:</p> let (tx, rx) = channel::new(); let (broadcast_tx, broadcast_rx) = broadcast_channel(); let forward_task = spawn(async move { loop { let result = rx.recv().await; match result { Ok(input) => { let output = get_output(input); broadcast_tx.send(output).unwrap(); } Err(_) => break, } } }); </code></pre> In this example, I use “watch” and “broadcast” channels from the Tokio crate.</p> If we would translate this to a functional asynchronous version, we get:</p> let (sender, receiver) = channel::new(); let (broadcast_sender, broadcast_receiver) = broadcast_channel(); let forward_task = spawn( async move { receiver .map(Result::ok) .filter_map(\|result\| result.ok()) .map(get_output) .forward(broadcast_sender) } ); </code></pre> So what happened during the translation?</p> Replace the channel receiver with an implementor of the trait Stream</code> from futures</code>. A stream</strong> is just something that produces data at possibly irregular intervals. A channel receiver is an example of this.</li> Replace the channel sender with a type implementing the Sink</code> trait from futures</code>. A sink</strong> is something that receives data, agnostic from the transport or channel used. An implementor of the Sink</code> is something in which you can put data and flush it.</li> Redirect</strong> the stream into the sink with forward</code>. This process could be seen as “flushing” the stream into the sink. However, the flush</code> name is already taken by the flush</code> method of the Sink</code> trait.</li> </ul> By shifting the focus from input, intermediate, and output variables to transformations with map</code>, we replaced N imperative variables in the loop by N-1 functional</strong> closures passed to map</code>. You could still argue that the variables did not really disappear, but they are now hidden in the closures and the code is more readable.</p> Remark: JavaScript frameworks for building web-apps usually call sink</code> a “signal” or “writable observable”. The gstremar-rs</code> crate also has “sinks” but they are not directly related to the “sinks” in futures</code></em></p> Reactive UI input</a></h3> Another place where functional asynchronous programming is useful is on the frontend.</p> An imperative version might look like this:</p> let mut target_temperature = 21.0; slider.on_slide(move \|new_t\| { if acceptable(new_t) { let new_target = some_op(new_t); target_temperature = new_target; } }); </code></pre> Although this particular example is small, writing large amounts of a large codebase in this style could introduce problems:</p> It introduces a lot of indentation. A developer who is new to the codebase might feel intimidated by the indentation. After careful consideration, he might decide to add his own logic on top of the existing code and then split the whole thing up into smaller chunks, thinking he improves readability, but actually breaking sequential readability.</li> By using an if</code> statement, you also create multiple branches. The reader or maintainer has to know that the else</code> branch is irrelevant in this particular case. A call to StreamExt::filter</code> already conveys the message that the true</code> branch is the only one that matters.</li> A maintainer also has to keep track of one more variable new_t</code> (the argument to the closure). The naming of intermediate variables (variables for data that appears before or after a computation) is hard, and names like old_t</code>, new_t</code>, or updated_t</code> are not helpful for the reader.</li> </ul> The imperative version can be translated to a functional version like this:</p> let mut target_temperature = MySink::new(21.0); slider.value .filter(acceptable) .map(some_op) .forward(target_temperature); </code></pre> The MySink::new(21.0)</code> is a call to the constructor of MySink</code>, an imaginary object that implements the Sink</code> trait.</p> Clear benefits</a></h3> Instead of exposing variables names for input, intermediate, and output variables, we omit them and focus on naming the transformations themselves. This way of dealing with computation is closer to how we communicate in natural language using verbs.</p> Another benefit of the functional approach is that it does not rely on a concrete type. If you are stuck in the middle of writing a module to provide a Stream</code> but you also have to write something that consumes it, you can just continue with the last one.</p> struct MyStream { // ... } impl MyStream { fn new() -> Self { unimplemented!() } } impl Stream for MyStream { type Item = i32; fn poll_next(self) -> Poll<Option<Self::Item>> { unimplemented!() } } </code></pre> You could then already start with another module that consumes it. But instead of directly depending on MyStream</code>, you can just depend on the Stream</code> trait. This way, you can write your code without having to know the implementation details of MyStream</code>.</p> struct MyConsumer { // ... } impl MyConsumer { fn consume(stream: impl Stream<Item = i32>) { // We can start already! // ... } } </code></pre> The important part is impl Stream<Item = i32></code>. This means that the consume</code> function can take any type that implements the Stream</code> trait and produces i32</code> items.</p> We have separated the problem into two levels of abstraction that can be dealt with independently and simultaneously:</p> The trait</strong> level: describes the invariants and properties of inputs and outputs. Working on this level in Rust is accomplished using the impl Trait</code> syntax.</li> The concrete type</strong> level: describes the transport, the speed, the efficiency, the logic. For a concrete type, you will have to either:</li> Pick an implementor from a public crate (like futures</code> or tokio</code>).</li> Write your own implementor of the Stream</code> trait. This is a bit more work, but it is not that hard. I will show you how to do this later on.</li> </ul> As of April 2025, the traits Stream</code> and Sink</code> are used universally by crates published on crates.io</a>. If you make use of these traits, which describe a common behavior, and implement them for your own types, you make your code more interoperable with the rest of the world.</p> Important</strong>: It is important to know while using Rust that it is not required to know everything about Pin</code> or Poll</code>. You can just use the high-level methods provided by the standard library and futures</code> crate.</p> Streams</a></h3> Relationship with iterators</a></h3> The main thing that is added in functional asynchronous programming is the Stream</code> trait. You are supposed to use it everywhere. There are other things of course, but this is the main concept that you will need.</p> So what is a stream</strong>? It is just something that implements the Stream</code> trait from the futures</code> crate. It nothing more than an asynchronous iterator.</p> A rough conceptual definition of a stream would be:</p> A function that returns multiple values at unpredictable times.</p> </blockquote> First, remember that the life-time of an iterator (a normal synchronous blocking one) looks like this:</p> T</th> create</th> iterate</th> yield</th></tr></thead> 1</td> (1..=10)</code></td> </td> </td></tr> 2</td> </td> next()</code></td> </td></tr> 3</td> </td> </td> Some(1)</code></td></tr> 4</td> </td> next()</code></td> </td></tr> 5</td> </td> …</td> </td></tr> 6</td> </td> next()</code></td> </td></tr> 7</td> </td> </td> None</code></td></tr> 8</td> </td> next()</code></td> </td></tr> 9</td> </td> </td> Some(2)</code></td></tr> </tbody></table> Notice that calling next</code> after the iterator yielded None</code> may result in a new Some</code>. If you do not want that, apply fuse</code> to the iterator to obtain a FusedIterator</code> that will keep yielding None</code> after the first None</code>.</p> The life-time of a stream/async iterator during usage looks like this:</p> T</strong></th> Creation</strong></th> Iteration</strong></th> Yielted</strong></th></tr></thead> 1</td> St::new()</code></td> </td> </td></tr> 2</td> </td> next()</code></td> </td></tr> 3</td> </td> await</code></td> </td></tr> 4</td> </td> </td> Some(1)</code></td></tr> 5</td> </td> next()</code></td> </td></tr> 6</td> </td> …</td> </td></tr> 7</td> </td> await</code></td> </td></tr> 8</td> </td> </td> Some(2)</code></td></tr> 9</td> </td> next()</code></td> </td></tr> 10</td> </td> </td> None</code></td></tr> </tbody></table> The lifecycle of an async iterator (stream) is longer than a normal iterator since it requires an await</code> before a value is yielded.</p> A FusedStream</code> is the async analogue of FusedIterator</code> and will yield None</code> after the first None</code>. In addition, it has a non-async method is_terminated</code> that says whether the stream is exhausted already.</p> pub trait FusedStream: Stream { fn is_terminated(&self) -> bool; } </code></pre> Usually a FusedStream</code> will yield Poll::Ready(None)</code> after the first Poll::Ready(None)</code> and it’s is_terminated</code> method will be positive. However, the implementor has the freedom to break these conventions.</p> Consuming streams</a></h3> Remark</strong>: As of April 2025, all the methods you need for streams are in StreamExt</code></a> from futures</code>. For the rest of this article, almost all Stream</code>-related methods come from this trait.</em></p> The simplest case would be the case where you just want to perform an operation for each element that is yielded by the stream. For this, you should use the for_each</code> method to act on each item in Some(item)</code>.</p> let mut x = 0; let fut = stream::repeat(1).take(3).for_each(\|item\| { x += item; future::ready(()) }); fut.await; assert_eq!(x, 3); </code></pre> Observe that the argument for the closure in for_each</code> does not take an Option</code>. The stream returned by for_each</code> is fused; it is an implementor of FusedStream</code>. A FusedStream</code> is a special type of stream that terminates as soon as one item yielded by the input stream is None</code>.</p> Important</strong>:</p> The futures that are evaluated while consuming the for_each</code> will not overlap in execution. They will happen sequentially.</li> The for_each</code> will not do any work, unless it is driven</strong> by an asynchronous executor. The reason is that streams are lazy</em>, just like normal iterators in Rust.</li> </ul> Very often we just have to apply one blocking, synchronous operation to every output item from an input stream and return a new stream with the mapped items. The map</code> stream operator is the right tool for this:</p> let stream = stream::iter(1..=3); let stream = stream.map(\|x\| x + 3); </code></pre> When the operation in the closure is asynchronous you should use then</code> (as in Future::then</code>).</p> let stream = stream::iter(1..=3); let stream = stream.then(\|x\| async move { convert(x).await }); </code></pre> Feel free to use async closures async \|_\| {}</code> or AsyncFn</code> in recent Rust versions. Asynchronous closures were only recently stabilized as of April 2025. Since I do not understand the implementation of async closures very well, I prefer to keep using the old syntax: \|x\| async move {}</code> for now. This syntax works better with older versions of Rust.</p> Remark</strong>: In previous major version releases of futures</code>, then</code> and map</code> were a single function. The crate futures-preview</code> an old version. Avoid reading the documentation of futures-preview</code> to prevent confusion. (Everything you need for the rest of this presentation is available in futures >= 0.3.31</code>.)</em></p> Stream test helpers</a></h3> While implementing your own streams (maybe not now but later on), you will run into situations where you need consume the streams as if you were a typical consumer. The futures</code> crate provides helpers for tests that are analogous to the ones in Iterator</code>. The only thing that distinguishes them is being operators on asynchronous iterators</strong>:</p> The following Stream</code> helpers / operators take a stream and perform some simple actions on it without changing the values:</p> skip_while</code>: Drops items from a stream while a condition is met (provided as a boolean closure on the items of the input stream).</li> peekable</code>: Adds a peek</code> method that can be used to preview a reference to the next item</strong> without consuming it (yet). This is useful when you don’t want to step too fast</em> through the stream.</li> take</code>: Simply skip a number of items from the beginning of the stream.</li> enumerate</code>: Adds an increasing index to the items of stream, starting from the beginning.</li> </ul> For example, you can use enumerate</code> as follows:</p> let stream = stream::iter(vec!['a', 'b', 'c']); let mut stream = stream.enumerate(); assert_eq!(stream.next().await, Some((0, 'a'))); assert_eq!(stream.next().await, Some((1, 'b'))); assert_eq!(stream.next().await, Some((2, 'c'))); assert_eq!(stream.next().await, None); </code></pre> Filtering streams</a></h3> You can filter a stream of numbers to only keep the even numbers as follows:</p> use std::future::ready; use futures::{stream, StreamExt}; let stream = stream::iter(1..=10); let events = stream.filter(\|x\| ready(x % 2 == 0)); </code></pre> Notice the ready</code> function. This function maps primitive Rust values into the async world</strong>. The output of ready</code> is a minimal Future</code> that can be moved: it is Unpin</code></strong>.</p> Remark</strong>: Don’t try to implement ready</code> yourself, just import it from std::future::ready</code>.</em></p> Boolean operators</a></h3> The futures</code> crate also provides analogues for the boolean operators shipped with the standard library Iterator</code> such as any</code>, all</code>, … :</p> let number_stream = stream::repeat(1).map(\|n\| n); let less_then_twenty = number_stream.all(\|i\| async move { i < 20 }); assert_eq!(less_then_twenty.await, true); </code></pre> Notice here that we don’t have to “pin” the less_then_twenty</code> stream, because Unpin</code> is not a requirement for all</code>.</p> Sinks</a></h3> Dual of streams</a></h3> Up until now we have only seen detailed usage of the Stream</code> trait. But the opposite of a stream, a “sink”, is also shipped by the futures</code> crate as the Sink</code> trait. A Sink</code> is something that receives data, agnostic from the transport</strong> or channel used.</p> The different life-cycle stages of a Sink</code> can be summarized as follows:</p> stage</th> name</th> method</th> meaning</th> remark</th></tr></thead> creation</td> new</td> </td> Initial state</td> </td></tr> send</td> ready</td> ready().await</code></td> Wait until cache ready</td> may be full</td></tr> send</td> start send</td> start_send(item)</code></td> Load into cache</td> not actual send</td></tr> send</td> flush</td> flush().await</code></td> Send items from cache</td> </td></tr> close</td> close</td> close().await</code></td> Close the Sink</code></td> not automatic</td></tr> </tbody></table> The analogue of the map function for streams StreamExt::map</code> for Sink</code>s is the sink operator SinkExt::with</code>. Instead mapping the output items of a stream, it applies a mapping function to all items that are going to be flushed into the sink.</p> If a Sink</code> becomes full easily and you depend on the concrete underlying type (such as for example a sink derived from a particular type of sender of a Tokio channel), you can allocate an extra buffer with StreamExt::buffer()</code> to cache elements that don’s fit in the sink.</p> Remark: The Sink</code> trait is not as common as the Stream</code> trait in the crates that I have used. It is, however, very easy to implement yourself.</em></p> Flushing a Stream</code> into a Sink</code></a></h3> A Sink</code> may appear in combination with a Stream</code>. In that case, it is possible to create a fully functional pipeline that takes a stream and flushes it into a sink. This is done with the forward</code> method of the Sink</code> trait.</p> let (output,_) = channel::new(); let output = PollSender::new(output); let input = stream::repeat(1).map(Ok); input.forward(output).await.unwrap(); </code></pre> Important:</p> StreamExt::forward</code> takes a TryStream</code> (items are Result</code>)</li> returns future of Result</code> (need to ignore error).</li> </ul> More Sink</code> operators</a></h3> The forward</code> method will also close the Sink</code> upon termination of the input stream. If you don’t want to close the Sink</code> after stream returned None</code>, use the sink operator SinkExt::send_all</code>.</p> When you have one input stream and know n</code> output sinks at compile-time</strong>, you can use StreamExt::fanout</code>. Otherwise you will need a mechanism to Clone</code> the input stream at run-time.</p> Splitting streams</a></h3> Collapsing an iterable of streams</a></h3> Given an iterable of streams, you can collapse the whole iterable into one stream with select_all</code>. This function will just emit the stream items from all the streams (assuming the iterator is finite) as they arrive.</p> A simple example would look like this:</p> let stream_a = stream::repeat(1); let stream_b = stream::repeat(2); let merged = stream::select_all([stream_a, stream_b]); </code></pre> In practice, you would typically pass large vectors, compile-time-sized arrays or other iterable collections to the select_all</code> function.</p>

Willem Vanhulle

About

2026-01-08T00:00:00+00:00

Professional</h2>
I am a mathematician and computer scientist</em> from Ghent, Belgium. I attempted physics, but became enamoured with fermenting bacteria</a> before finishing. Since 2020, I have worked in scientific and tech start-ups as a software engineer. In 2025, I gathered my experience and taught a Rust course</a> for intermediate developers.</p>
In the past year I broadened my horizon to include C++ in my expertise and help companies deal with their large legacy codebases.</p>
For a more complete overview, see my CV</a>.</p>

Personal</h2>
I organise SysGhent</a>, an open community for systems programmers. It’s a group of people who finished their studies, but miss learning and we do lectures and meet-ups. Life is too short to sit only behind your computer.</p>
I maintain a tool for learning the Nu scripting language: `nu-lint</code></a>.</p>`
My latest open-source projects are hosted at Codeberg</a></li> Recently, GitHub became slow and bloated, but you can still find there at GitHub</a></li> </ul> </div> Contact</h2> Are you interested in hiring me or working together? Reach out!</p> Things I could certainly help your with:</p> Rust training</li> Distributed systems</li> Formal verification</li> Embedded / IoT</li> Migrations from C/C++ to Rust</li> Anything else that you currently need …</li> </ul> Email</a> | Phone</a> | LinkedIn</a> </nav>

Making generators 2025-05-08T00:00:00+00:00 Simple generators</a></h2> Iterators</a></h3> In functional programming, iterators replace loops. Rust provides many helper methods (called adapter methods</em>) for iterators such as map</code>, filter</code>, step_by</code>, … . But to apply this style of programming, you need some base iterators to start with.</p> The base (or leaf) iterators are the ones that are actually important. They are provided by the core language or a foundational user crate. Usually, it is a bad idea to implement your own iterators.</p> In case you decide to implement a new iterator anyway, have a look at the definition of an iterator:</p> pub trait Iterator { type Item; fn next(&mut self) -> Option<Self::Item>; } </code></pre> Blocking generators</a></h3> Although you could directly implement next</code> for your own data-types, it might be more straightforward to use a generator</strong> to create an iterator for you. Generators are functions that output an anonymous type that implements the Iterator</code> trait. The body of a generator has yield X</code> statements that represent the result of an invocation to next()</code> being Some(X)</code>.</p> Remark</strong>: There is nothing special or new about generators in Rust. They have existed, for example, in JavaScript for many years.</em></p> A good crate in Rust for writing generators is genawaiter</code></a>.</p> let generator = gen!({ yield_!(10); }); let xs: Vec<_> = generator.into_iter().collect(); assert_eq!(xs, [10]); </code></pre> Using this crate, the generator</code> variable is actually more than just a generator</em> (something that can be converted into an Iterator</code>). It is also a coroutine. See other posts on this blog to know more about coroutines in Rust.</p> Remark: The gen!</code> and yield_!</code>-macros will become built-in the core Rust language in the coming months. The gen!</code> simply becomes the gen</code> keyword for code blocks. Inside gen</code>-blocks you can use yield</code>. For now, you need nightly to use this.</em></p> Simple async generators</a></h3> The iterators generated by the previous kind of generators is blocking</em> (or synchronous ). The asynchronous (non-blocking) variant of a blocking iterator is a stream</strong> (an asynchronous iterator).</p> You can just keep using the genawaiter</code> crate and add await</code>-points in the body of your gen!</code> generator definition. You need to, however, enable the futures03</code> feature.</p> (From the documentation)</p> async fn async_one() -> i32 { 1 } async fn async_two() -> i32 { 2 } let generator = gen!({ let one = async_one().await; yield_!(one); let two = async_two().await; yield_!(two); }); let items: Vec<_> = stream.collect().await; assert_eq!(items, [1, 2]); </code></pre> An alternative is the async-stream</code> crate (which has been updated more recently). The generators written with its stream!</code> macro are always asynchronous streams (in contrast to genawaiter</code> which also supports iterators).</p> Remark: Asynchronous generators will be included in stable rust in the coming months. As of May 2025 you still need to switch to a nightly compiler version and enable unstable features.</em></p> Maintainable generators</a></h2> Generator state</a></h3> While creating generators and putting yield statements, you will quickly run into very complex code. Code making use of yield</code> may be hard to maintain. You will need a place to store the state of the generator. In the brute-force approach you just add local variables outside the main loop of your generator body.</p> For example, you could have something like:</p> let generator = gen!({ let mut state = Some(0); loop { do_something(); state.update(); yield_!(state.method()); } }); </code></pre> When state becomes bigger, it is time to switch to another approach.</p> A more structured construction</a></h3> The most straightforward alternative for async generatorss is to use the futures::stream::unfold</code> function. This function stores the state explicitly in its first argument and updates it incrementally with a closure (returning a Future</code>).</p> use futures::{stream, StreamExt}; let stream = stream::unfold(0, |state| async move { if state <= 2 { let next_state = state + 1; let yielded = state * 2; Some((yielded, next_state)) } else { None } }); let result = stream.collect::<Vec<i32>>().await; assert_eq!(result, vec![0, 2, 4]); </code></pre> A synchronous version of this (for normal Iterator</code>s) can be found in the crate itertools</code>: itertools::unfold</code></a>.</p> An example from the crate docs:</p> let mut fibonacci = unfold((1u32, 1u32), |(x1, x2)| { // Attempt to get the next Fibonacci number let next = x1.saturating_add(*x2); // Shift left: ret <- x1 <- x2 <- next let ret = *x1; *x1 = *x2; *x2 = next; // If addition has saturated at the maximum, we are finished if ret == *x1 && ret > 1 { None } else { Some(ret) } }); itertools::assert_equal(fibonacci.by_ref().take(8), vec![1, 1, 2, 3, 5, 8, 13, 21]); </code></pre> Functional combinators</a></h3> One problem with unfold</code> is that less well-suited for scenarios in which you need to combine or split several streams/iterators, for example while making your own functional combinators. The constructor unfold</code> seems to be most appropriate when you need to create iterators or streams from scratch</strong>.</p> Writing your own combinators gives you the tools to recombine streams</em> in a more functional or declarative way. Have a look at the combinators defined in futures::stream</code></a> to see what a common combinator implementation looks like.</p> Remark: See other posts about combinators for information on how to build your own declarative combinators</strong> such as custom variants on merge</code> or flatten</code>.</em></p> Building stream combinators 2025-04-16T00:00:00+00:00 Introduction</a></h2> What if you know how to use streams, but you need some kind of functionality that is not in the StreamExt</code> trait from futures</code> or the standard library?</p> In that case, you might try an imperative approach and create the stream using the unfold</code> function. This is a solution for simple cases, but it does not generalise well. Disambiguating between different parts of intermediate stream state quickly becomes difficult.</p> You might also want to try using imperative design patterns like loops, channels, spawns and generic functions. This approach quickly becomes unmaintainable because of its complexity.</p> Invalidation through moves</a></h2> Remark</strong>: the following sections will be about Unpin</code>. I wrote about it because I needed it later on to understand better how to construct combinators. You can skip this chapter if you want (or look at the official documentation).</em></p> The physical location of variables used in your code may move throughout the lifetime of your program. In Rust, for example, it is common to move a variable through an assignment. A variable that is called by value by a function is moved</strong> (literally and conceptually) into the the function. The function takes conceptual ownership. As far as I know, this is called the move semantics</strong> of Rust.</p> However, through this conceptual/semantical moving process, a physical move of the data in the registers of the CPU or other parts of memory may also occur. This is dangerous when the data being moved is self-referencing.</p> For more information, see std::pin</code></a>.</p> Clearing up Unpin</code></a></h3> The confusing aspect of Pin</code> and Unpin</code> in Rust, is that it is not Pin</code> which is the first that you should understand, but it is Unpin</code>.</p> The naming of Unpin</code> makes it seem like it is some counterpart to Pin</code>. However, it is not.</p> Unpin</code> is an auto-trait which means that the compiler derives Unpin</code> for everything that it deems safe to move</strong>. This is done at compile-time and behind the scene by the compiler for every “auto-trait”.</p> Cannot be moved / !Unpin</code></a></h3> Anything that looks like it cannot be moved by the compiler, will be marked automatically as !Unpin</code>, not Unpin</code>, or un-moveable. The reasoning by the compiler is that it necessary to detect when some kind of data type would be invalidated by a move and prevent dangerous actions by users (programmers implementing async functionality).</p> Examples of !Unpin</code> Rust data-types:</p> generators:</p> normal generators written with gen</code>-blocks or macros</li> async generators written with the macro async_stream::stream! {}</code> (implementing the Stream</code> trait)</li> </ul> </li> self-referencing data that is code-generated by the compiler</strong></p> state machines generated by async {}</code> blocks</li> … ?</li> </ul> </li> self-referencing user data structures:</p> naive trees</li> strings with slices</li> </ul> </li> types with a manual PhantomPinned</code> field</p> </li> </ul> Important</strong>: The reason that async {}</code> blocks are !Unpin</code> is that the compiler is lazy and does not analyze such blocks automatically to check whether they are Unpin</code>.</p> Purpose of data-type Pin</code></a></h3> Pin</code> as a data-type has only one requirement: it’s inner type should implement Deref</code>. In other words, Pin</code> is a wrapper around pointer / reference-like datatypes.</p> The concrete type Pin</code> is, in fact, not a real physical type. It does not represent a different location or address in memory. It is merely a Rust compiler construct that manifests itself as a type available to the users.</p> The Pin</code> type is essentially a contract for pointer-like types, maintained by two methods that require the pointed-to (also called “pointee”) to implement Unpin</code>:</p> A constructor Pin::new()</code></strong>: takes any owned Unpin</code> object (safe-to-move, not !Unpin</code>) and takes ownership of it. There is no way to get ownership back. You may, however, implement a Drop</code> implementation which will be run on de-allocation of Pin</code>.</li> A mutable getter Pin::get_mut()</code></strong>: allows you get mutable access to the contained, pinned, Unpin</code> value. This allows you to still call methods with the &mut Self</code> signature on the pinned data.</li> </ul> The reason that the get_mut</code> method requires the pointee to be Unpin</code> is that mutable access through a mutable reference can be used to move the content of the Pin</code> with a function like std::mem::replace</code>. This could invalidate any pinned otherwise unmoveable !Unpin</code> type.</p> Metaphor</strong></th> Type state</strong></th> Ownership event</strong></th></tr></thead> undressed</em></td> Type</code></td> moveable / free</td></tr> dress-up</em></td> Pin::new(Type)</code></td> give up ownership</td></tr> dressed-up</em></td> Pin<Type></code></td> stuck in memory</td></tr> dress-down</em></td> Pin::new(Type).get_mut()</code></td> acquire edit access</td></tr> </tbody></table> Conclusion:</p> Pin</code> does not pin anything physically at run-time, but relies on the auto-trait !Unpin</code> to prevent dangerous moves at run-time.</p> </blockquote> Convert !Unpin</code> into Unpin</code></a></h3> Almost all primitives in Rust are Unpin</code>. Whenever you encounter something that is not Unpin</code> (and has not non-'static</code> references), you can just allocate it on the heap with the function Box::pin</code>. The Box::pin</code> is shorthand for Pin::new(Box::new())</code>.</p> The result of the Box::pin</code> is a combination:</p> The heap-allocated pinned object will not be moved by any code generated by the Rust compiler;</li> We can still drop the pinned object and the Drop</code> implementations will run.</li> </ul> Let’s take the following as an example of something that is !Unpin</code>.</p> struct MapFut<Fut> where Fut: Future { fut: Fut, ... } </code></pre> A common approach when handling extensions or combinators of futures which are themselves futures, is by projecting</strong> Pin<&mut Self></code>into &mut Self</code>. The projection is usually called this</code> (the self</code> keyword is reserved).</p> impl<Fut> Future for MapFut<Fut> { fn poll(self: Pin<&mut Self>) -> Poll { // We need to do something with `self.fut`. // We need `Pin<&mut Self> -> &mut Self`. // The following will not compile, because `Self: !Unpin`. let mut this = self.get_mut(); unimplemented!() } } </code></pre> In this example the user struct MapFut</code> is !Unpin</code>, so get_mut</code> cannot be used. The get_mut</code> function requires Self: Unpin</code> as mentioned in the previous section.</p> The simplest solution is to refine the definition of MapFut</code> as follows:</p> struct MapFut<Fut> where Fut: Future { fut: Pin<Box<Fut>>, ... } </code></pre> Now fut</code> is a pinned pointer to a location on the heap.</p> Unsafe Pin</code> methods (optional)</a></h3> Until now I explicitly avoided mentioning or using the unsafe methods of the Pin</code> type. Beside Pin::new()</code> and Pin::get_mut()</code>, Pin</code> also has a few unsafe</code> counterparts.</p> You might have been tempted by the Rust compiler to use the unsafe</code> methods because they do not have an Unpin</code> constraint. But if you go in that direction, you effectively disable automatic and important checks by the compiler at run-time.</p> The Ready</code> future</a></h3> Anything can be turned into a future using the Ready</code> future. This future takes the source and puts it inside an Option</code>.</p> pub struct Ready<T>(Option<T>); </code></pre> This future never has to wake up, because it yields a value immediately and is read on the first poll</code> call. This is why the context (and its waker) are ignored.</p> impl<T> Future for Ready<T> { type Output = T; fn poll(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<T> { Poll::Ready(self.0.take().expect("Ready polled after completion")) } } </code></pre> To use this future, you can directly call the constructor Ready::new()</code> or you can call ready()</code>:</p> assert_eq!(1, ready(1).await)`; </code></pre> Remark: .await</code> automatically converts everything in a future with IntoFuture</code>, so it might not even be necessary to use Ready</code> explicitly.</em></p> Simple stream combinators</a></h2> Definition of a stream</a></h3> A stream is an future that may be polled more than once and yield more than one Poll::Ready</code> value.</p> The official definition is a trait:</p> pub trait Stream { type Item; fn poll_next( self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll<Option<Self::Item>>; } </code></pre> Notice the similary with the Future</code> trait.</p> Important: As of April 2025, the Stream</code> trait is not yet in stable Rust.</em></p> Futures as streams with Once</code></a></h3> Every future Fut</code> can be converted into a stream by returning Some(Fut::Output)</code>:</p> pub struct Once<Fut> { future: Option<Fut> } impl<Fut: Future> Stream for Once<Fut> { type Item = Fut::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { let mut this = self.project(); let v = match this.future.as_mut().as_pin_mut() { Some(fut) => ready!(fut.poll(cx)), None => return Poll::Ready(None), }; this.future.set(None); Poll::Ready(Some(v)) } } </code></pre> Here, ready!</code> converts Poll</code> into Option</code>.</p> Functional building block Map</code></a></h3> Let’s look at a simple example from the “semi-standard” library crate futures</code>: the map</code> combinator. This combinator just maps the items of an input stream and returns an new output stream (while consuming the input stream.)</p> The combinator’s source code looks like this:</p> pub struct Map<St, F> { #[pin] stream: St, f: F, } impl<St, F> Stream for Map<St, F> where St: Stream, F: FnMut<St::Item>, { type Item = F::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { ... } } </code></pre> Ignore the field attribute pin</code> for now.</p> The definition of map</code></a> and all the behaviour that we expect from a map</code> function essentially boils down to storing two things:</p> The original input stream.</li> The function or closure that maps elements of the input stream.</li> </ul> Then the authors of futures</code> had to implement the single required method of the Stream</code> trait: poll_next</code>:</p> impl<St, F> Stream for Map<St, F> where St: Stream, F: FnMut<St::Item>, { type Item = F::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> { let mut this = self.project(); let res = ready!(this.stream.as_mut().poll_next(cx)); Poll::Ready(res.map(|x| this.f.call_mut(x))) } } </code></pre> At the beginning of the poll_next</code> function, only a Pin<&mut Self></code> is given. It has to be “undressed” to a mutable reference to the state object Map</code>. This happens through a process called projection</strong>, which is essentially just calling get_mut</code> on Pin</code>.</p> Projecting with pin_project</code> (optional)</a></h3> Almost any library I encountered uses the pin_project</code> crate which provides the #[pin]</code> field attribute that I asked you to ignore. The pin_project</code> crate takes care of two things:</p> You don’t have to call Box::pin</code> yourself or pin_mut!</code> inside the constructor function of your aggregated stream.</li> You get a project</code> function which allows you to call as_mut()</code> (equivalent to get_mut()</code>).</li> </ul> Remark</strong>: Internally (if I read the code correctly), pin_project</code> makes use of several unsafe</code> function calls of the Pin</code> type. This is not that strange, considering that most of the standard library is written in unsafe</code>. On the other hand, using Box::pin</code> will not have a significant performance impact and it does not require a macro from an external crate.</p> Stream combinators</a></h2> I will now focus on more complicated stream combinators. These are functions that take one or more input streams and output an output stream. I will call such higher-order functions stream combinators</strong> from now on.</p> Remark: The futures-rx</code></a> crate is a good crate to look for if you want to save time and just use combinators made by someone else.</em></p> Flattening nested streams</a></h3> The futures</code> crate has several types of flatten functions for nested streams. Nested streams are Rust data types S</code> with a trait bound S: Stream<Item: Stream></code>.</p> You could separate flatten combinators in two categories: sequential or concurrent.</p> Sequential flatten combinators will never yield values from multiple inner streams at the same time.</p> flatten</code>: flattens the outer stream by pasting the output from the inner streams consecutively. This is usually not what you want, since most streams are infinite and you don’t want to block the outer stream.</li> A “forgetful” flatten (also called switch</code> in RxJs): a kind of flatten of nested streams that only polls the most recent stream that arrived on the outer stream. This is implemented by switch_map</code></a> in futures-rx</code>.</li> </ul> Concurrent flatten combinators will combine values from multiple inner streams:</p> Concurrent flatten_unordered(None)</code>: flattens by merging as many inner streams as possible, as they arrive on the outer stream. This might seems like a useful function, but it often not what you want.</li> Buffered concurrent flatten_unordered(Some(N))</code>: flattens up-to N different inner streams.</li> </ul> Keep in mind that you can also use the select_all</code> function from futures::stream</code> to flatten in the situation where your outer stream is not a real stream but a finite iterable of streams.</p> A stream of Arc</code>s</a></h3> Sometimes you might want to use the items yielded by a single stream in several places simultaneously. You could wrap all the items yielded by the input stream inside a shared reference Arc</code>. The disadvantage of this approach is that you need (a limited form of) garbage cleaning by reference counting.</p> The futures-rx</code> crate exposes a share</code> method that can be used for splitting an input stream.</p> Looking at the docs</a>, there is an example:</p> use futures::{stream::{StreamExt, self}, future::join}; use futures_rx::{Notification, RxExt}; let stream = stream::iter(0..=3); let stream = stream.share(); let sub_stream_a = stream.clone().map(|event| *event); let sub_stream_b = stream.clone().map(|event| *event); assert_eq!((vec![0, 1, 2, 3], vec![0, 1, 2, 3]), join(sub_stream_a.collect::<Vec<_>>(), sub_stream_b.collect::<Vec<_>>()).await); </code></pre> The event</code>s yielded by the shared stream seem to be wrappers around Arc<usize></code>. Since usize</code> is Copy</code>, *event</code> is treated as &usize</code> and the referenced usize</code> automically cloned.</p> Remark: Maybe I am wrong?</em></p> A stream of clones</a></h3> The share</code> operator on streams from futures-rx</code> seems to put all values on the input stream inside a reference-counted Arc</code>.</p> I wanted to derive several output streams from the input stream that yielded clones directly without reference counting.</p> Remark: Maybe I could just have used share</code> and then cloned every element?</em></p> I decided to implement a separate stream cloning trait in a crate clone-stream</code></a>. The high-level API of this crate is quite simple. It is just a call to fork</code> and then stream.clone()</code>:</p> use clone_stream::ForkStream; use futures::{FutureExt, StreamExt, stream}; let non_clone_stream = stream::iter(0..10); let clone_stream = non_clone_stream.fork(); let mut cloned_stream = clone_stream.clone(); </code></pre> Remark: There exist a few other crates that are similar.</em></p> For more information about the clone-stream</code> crate, see my presentation at EuroRust 2025</a>.</p> Role of coroutines 2025-04-01T00:00:00+00:00 In some other posts on this site, you will find ways to create streams from scratch and how to combine them. This post will be about the relationship between the concept of a Stream</code> (or asynchronous iterator) and the other, more familiar, functions present in most programming languages.</p> Most of this post was inspired by a post by without.boats</a>.</p> Simple coroutines</a></h2> Concept of a coroutine</a></h3> Normal functions return output (immediately). They do it only once (or never).</p> A coroutine</strong> is a special kind functions that can:</p> be suspended multiple times</li> be resumed multiple times</li> return once</li> </ul> More specifically, at runtime, coroutines go through a process (with specific terminology):</p> When a coroutine suspends, it yield</strong>s a value to the caller. This is a kind of intermediate return value.</li> After observing (or ignoring) the yielded value, the caller can safely forget about the suspended coroutine (temporarily) and continue with other functions.</li> Later the caller can return to this suspended coroutine. The caller needs to resume the suspended coroutine to wake it up. This step is called resumption</strong>. For resumption some resumption data may need to be provided.</li> </ol> These steps may repeat forever or until the coroutine ends by returning. Returning is distinct from yielding, since it is final. The return value is the last value that can be observed by the caller.</p> Remark</strong>: Coroutines are used internally by the Rust compiler while compiling asynchronous code. The compiler implements a form of “stack-less” co-routines for async {}</code> code-blocks. These blocks are compiled implicitly into coroutines that yield at every await</code>-point.</em></p> Directly constructing coroutines</a></h3> The Coroutine</code> trait definition is an extension of the Future</code> trait:</p> pub trait Coroutine<Resume = ()> { type Yield; type Return; fn resume( self: Pin<&mut Self>, resumption: Resume, ) -> CoroutineState<Self::Yield, Self::Return>; } </code></pre> Notice that we need Pin</code>, similarly to Future</code>. This is because coroutines may be self-referential. The resume</code> function should only be called on coroutines that may move (are Unpin</code>). The reason is probably that they should extend the behaviour of Future</code>s which do require Pin</code>.</p> Important</strong>: The Coroutine</code> trait in Rust is unstable and only available on nightly as of April 2025.</em></p> If you look carefully at the Coroutine</code> trait you could see that (in pseudo-code):</p> type Future<Output> = Coroutine< Resume = Context, Yield = (), Return = Output >; </code></pre> More precisely, a future is a coroutine that yields nothing when suspended. A future needs a Context</code> (containing a Waker</code>) to be resumed or woken.</p> Remark</strong>: The resumption data is provided by a asynchronous run-time to schedule resume</code>s in an efficient way.</em></p> Example of a coroutine</a></h3> The Rust docs contain an example of a coroutine. The coroutine does not need any resumption data, but it yields a number and returns a string on completion:</p> let mut coroutine = #[coroutine] || { yield 1; "foo" }; </code></pre> To use this coroutine</code>, we have to provide an initial chunk of resumption data. By default this is the empty tuple ()</code>. The resumption data is passed to the resume</code> function and used to anticipate the first yield. The first (and last) yield is yield 1</code>.</p> match Pin::new(&mut coroutine).resume(()) { CoroutineState::Yielded(1) => {} _ => panic!("unexpected return from resume"), } </code></pre> The next time resume</code> is called, no yield is encountered and the final return value is returned (a string).</p> match Pin::new(&mut coroutine).resume(()) { CoroutineState::Returned("foo") => {} _ => panic!("unexpected return from resume"), } </code></pre> If our coroutine was a Future</code>, then resume</code> would expect a Context</code> with a Waker</code>.</p> Classification of coroutines</a></h2> Reflecting on the concepts of an iterator, future and stream, we can say that:</p> An iterator</strong> is coroutine that yields an Option</code>.</li> A future</strong> is a coroutine that resumes with a Waker</code>.</li> A stream</strong> is an iterator that resumes with a Waker</code> and yields an Option</code>.</li> </ul> Coroutines are a generalisation of these cases, which can be laid-out in a table:</p> </th> YIELDS</em></th> RESUMES</em></th> RETURNS</em></th></tr></thead> Iterator</code></td> Option</code></td> !</code></td> !</code></td></tr> Future</code>, AsyncFn</code></td> ()</code></td> Waker</code></td> Any</code></td></tr> Stream</code></td> Option</code></td> Waker</code></td> !</code></td></tr> Coroutine</code></td> Any</code></td> Any</code></td> Any</code></td></tr> </tbody></table> In this table, the !</code> symbol stands for never</code>, the type that does not have any runtime value. In other words, never</code> is not constructible. It is used often as the return time of non-terminating functions like infinite loops.</p> For a practical introduction to coroutines in Rust, I recommend Asynchronous Programming in Rust</a>.</p> Functional async 2025-03-18T00:00:00+00:00 Introduction</a></h2> Let’s get started with some definitions you may have heard about already.</p> Declarative programming</strong> is when you write code and make sure that the inputs and outputs of sub-modules behave predictably according to fixed invariants. Logic or constraint solver languages like Prolog or ManyWorlds</a> (written by my friend Jo Devriendt</a>) are part of this family.</p> Functional programming</strong> is building the majority of your code from well-defined and side-effect-free functions. It is a sub-set of declarative programming. Languages like ML, Haskell, OCaml, Lisp are part of this family. They are all extensions of Lambda Calculus.</p> Asynchronous programming</strong> is a kind of programming where you not only have functions that evaluate immediately, but there are also functions that may evaluate in the future</em>. Such functions are called asynchronous functions. JavaScript, C#, and Rust are examples.</p> In the following, I will show how declarative, functional, and asynchronous programming can be combined in the Rust programming language with the semi-standard library crate futures</code>.</p> Use-cases</a></h2> Normal imperative asynchronous programming contains a combination of the async</code>, await</code> keywords with imperative keywords like while</code>, loop</code>, or continue</code>.</p> To illustrate where functional asynchronous programming would be useful, I will give a few examples.</p> Channels</a></h3> One approach that you might take when dealing with different sub-modules in a large project is using channels to communicate between the different modules.</p> This could look like this:</p> let (tx, rx) = channel::new(); let (broadcast_tx, broadcast_rx) = broadcast_channel(); let forward_task = spawn(async move { loop { let result = rx.recv().await; match result { Ok(input) => { let output = get_output(input); broadcast_tx.send(output).unwrap(); } Err(_) => break, } } }); </code></pre> In this example, I use “watch” and “broadcast” channels from the Tokio crate.</p> If we would translate this to a functional asynchronous version, we get:</p> let (sender, receiver) = channel::new(); let (broadcast_sender, broadcast_receiver) = broadcast_channel(); let forward_task = spawn( async move { receiver .map(Result::ok) .filter_map(|result| result.ok()) .map(get_output) .forward(broadcast_sender) } ); </code></pre> So what happened during the translation?</p> Replace the channel receiver with an implementor of the trait Stream</code> from futures</code>. A stream</strong> is just something that produces data at possibly irregular intervals. A channel receiver is an example of this.</li> Replace the channel sender with a type implementing the Sink</code> trait from futures</code>. A sink</strong> is something that receives data, agnostic from the transport or channel used. An implementor of the Sink</code> is something in which you can put data and flush it.</li> Redirect</strong> the stream into the sink with forward</code>. This process could be seen as “flushing” the stream into the sink. However, the flush</code> name is already taken by the flush</code> method of the Sink</code> trait.</li> </ul> By shifting the focus from input, intermediate, and output variables to transformations with map</code>, we replaced N imperative variables in the loop by N-1 functional</strong> closures passed to map</code>. You could still argue that the variables did not really disappear, but they are now hidden in the closures and the code is more readable.</p> Remark: JavaScript frameworks for building web-apps usually call sink</code> a “signal” or “writable observable”. The gstremar-rs</code> crate also has “sinks” but they are not directly related to the “sinks” in futures</code></em></p> Reactive UI input</a></h3> Another place where functional asynchronous programming is useful is on the frontend.</p> An imperative version might look like this:</p> let mut target_temperature = 21.0; slider.on_slide(move |new_t| { if acceptable(new_t) { let new_target = some_op(new_t); target_temperature = new_target; } }); </code></pre> Although this particular example is small, writing large amounts of a large codebase in this style could introduce problems:</p> It introduces a lot of indentation. A developer who is new to the codebase might feel intimidated by the indentation. After careful consideration, he might decide to add his own logic on top of the existing code and then split the whole thing up into smaller chunks, thinking he improves readability, but actually breaking sequential readability.</li> By using an if</code> statement, you also create multiple branches. The reader or maintainer has to know that the else</code> branch is irrelevant in this particular case. A call to StreamExt::filter</code> already conveys the message that the true</code> branch is the only one that matters.</li> A maintainer also has to keep track of one more variable new_t</code> (the argument to the closure). The naming of intermediate variables (variables for data that appears before or after a computation) is hard, and names like old_t</code>, new_t</code>, or updated_t</code> are not helpful for the reader.</li> </ul> The imperative version can be translated to a functional version like this:</p> let mut target_temperature = MySink::new(21.0); slider.value .filter(acceptable) .map(some_op) .forward(target_temperature); </code></pre> The MySink::new(21.0)</code> is a call to the constructor of MySink</code>, an imaginary object that implements the Sink</code> trait.</p> Clear benefits</a></h3> Instead of exposing variables names for input, intermediate, and output variables, we omit them and focus on naming the transformations themselves. This way of dealing with computation is closer to how we communicate in natural language using verbs.</p> Another benefit of the functional approach is that it does not rely on a concrete type. If you are stuck in the middle of writing a module to provide a Stream</code> but you also have to write something that consumes it, you can just continue with the last one.</p> struct MyStream { // ... } impl MyStream { fn new() -> Self { unimplemented!() } } impl Stream for MyStream { type Item = i32; fn poll_next(self) -> Poll<Option<Self::Item>> { unimplemented!() } } </code></pre> You could then already start with another module that consumes it. But instead of directly depending on MyStream</code>, you can just depend on the Stream</code> trait. This way, you can write your code without having to know the implementation details of MyStream</code>.</p> struct MyConsumer { // ... } impl MyConsumer { fn consume(stream: impl Stream<Item = i32>) { // We can start already! // ... } } </code></pre> The important part is impl Stream<Item = i32></code>. This means that the consume</code> function can take any type that implements the Stream</code> trait and produces i32</code> items.</p> We have separated the problem into two levels of abstraction that can be dealt with independently and simultaneously:</p> The trait</strong> level: describes the invariants and properties of inputs and outputs. Working on this level in Rust is accomplished using the impl Trait</code> syntax.</li> The concrete type</strong> level: describes the transport, the speed, the efficiency, the logic. For a concrete type, you will have to either:</li> Pick an implementor from a public crate (like futures</code> or tokio</code>).</li> Write your own implementor of the Stream</code> trait. This is a bit more work, but it is not that hard. I will show you how to do this later on.</li> </ul> As of April 2025, the traits Stream</code> and Sink</code> are used universally by crates published on crates.io</a>. If you make use of these traits, which describe a common behavior, and implement them for your own types, you make your code more interoperable with the rest of the world.</p> Important</strong>: It is important to know while using Rust that it is not required to know everything about Pin</code> or Poll</code>. You can just use the high-level methods provided by the standard library and futures</code> crate.</p> Streams</a></h3> Relationship with iterators</a></h3> The main thing that is added in functional asynchronous programming is the Stream</code> trait. You are supposed to use it everywhere. There are other things of course, but this is the main concept that you will need.</p> So what is a stream</strong>? It is just something that implements the Stream</code> trait from the futures</code> crate. It nothing more than an asynchronous iterator.</p> A rough conceptual definition of a stream would be:</p> A function that returns multiple values at unpredictable times.</p> </blockquote> First, remember that the life-time of an iterator (a normal synchronous blocking one) looks like this:</p> T</th> create</th> iterate</th> yield</th></tr></thead> 1</td> (1..=10)</code></td> </td> </td></tr> 2</td> </td> next()</code></td> </td></tr> 3</td> </td> </td> Some(1)</code></td></tr> 4</td> </td> next()</code></td> </td></tr> 5</td> </td> …</td> </td></tr> 6</td> </td> next()</code></td> </td></tr> 7</td> </td> </td> None</code></td></tr> 8</td> </td> next()</code></td> </td></tr> 9</td> </td> </td> Some(2)</code></td></tr> </tbody></table> Notice that calling next</code> after the iterator yielded None</code> may result in a new Some</code>. If you do not want that, apply fuse</code> to the iterator to obtain a FusedIterator</code> that will keep yielding None</code> after the first None</code>.</p> The life-time of a stream/async iterator during usage looks like this:</p> T</strong></th> Creation</strong></th> Iteration</strong></th> Yielted</strong></th></tr></thead> 1</td> St::new()</code></td> </td> </td></tr> 2</td> </td> next()</code></td> </td></tr> 3</td> </td> await</code></td> </td></tr> 4</td> </td> </td> Some(1)</code></td></tr> 5</td> </td> next()</code></td> </td></tr> 6</td> </td> …</td> </td></tr> 7</td> </td> await</code></td> </td></tr> 8</td> </td> </td> Some(2)</code></td></tr> 9</td> </td> next()</code></td> </td></tr> 10</td> </td> </td> None</code></td></tr> </tbody></table> The lifecycle of an async iterator (stream) is longer than a normal iterator since it requires an await</code> before a value is yielded.</p> A FusedStream</code> is the async analogue of FusedIterator</code> and will yield None</code> after the first None</code>. In addition, it has a non-async method is_terminated</code> that says whether the stream is exhausted already.</p> pub trait FusedStream: Stream { fn is_terminated(&self) -> bool; } </code></pre> Usually a FusedStream</code> will yield Poll::Ready(None)</code> after the first Poll::Ready(None)</code> and it’s is_terminated</code> method will be positive. However, the implementor has the freedom to break these conventions.</p> Consuming streams</a></h3> Remark</strong>: As of April 2025, all the methods you need for streams are in StreamExt</code></a> from futures</code>. For the rest of this article, almost all Stream</code>-related methods come from this trait.</em></p> The simplest case would be the case where you just want to perform an operation for each element that is yielded by the stream. For this, you should use the for_each</code> method to act on each item in Some(item)</code>.</p> let mut x = 0; let fut = stream::repeat(1).take(3).for_each(|item| { x += item; future::ready(()) }); fut.await; assert_eq!(x, 3); </code></pre> Observe that the argument for the closure in for_each</code> does not take an Option</code>. The stream returned by for_each</code> is fused; it is an implementor of FusedStream</code>. A FusedStream</code> is a special type of stream that terminates as soon as one item yielded by the input stream is None</code>.</p> Important</strong>:</p> The futures that are evaluated while consuming the for_each</code> will not overlap in execution. They will happen sequentially.</li> The for_each</code> will not do any work, unless it is driven</strong> by an asynchronous executor. The reason is that streams are lazy</em>, just like normal iterators in Rust.</li> </ul> Very often we just have to apply one blocking, synchronous operation to every output item from an input stream and return a new stream with the mapped items. The map</code> stream operator is the right tool for this:</p> let stream = stream::iter(1..=3); let stream = stream.map(|x| x + 3); </code></pre> When the operation in the closure is asynchronous you should use then</code> (as in Future::then</code>).</p> let stream = stream::iter(1..=3); let stream = stream.then(|x| async move { convert(x).await }); </code></pre> Feel free to use async closures async |_| {}</code> or AsyncFn</code> in recent Rust versions. Asynchronous closures were only recently stabilized as of April 2025. Since I do not understand the implementation of async closures very well, I prefer to keep using the old syntax: |x| async move {}</code> for now. This syntax works better with older versions of Rust.</p> Remark</strong>: In previous major version releases of futures</code>, then</code> and map</code> were a single function. The crate futures-preview</code> an old version. Avoid reading the documentation of futures-preview</code> to prevent confusion. (Everything you need for the rest of this presentation is available in futures >= 0.3.31</code>.)</em></p> Stream test helpers</a></h3> While implementing your own streams (maybe not now but later on), you will run into situations where you need consume the streams as if you were a typical consumer. The futures</code> crate provides helpers for tests that are analogous to the ones in Iterator</code>. The only thing that distinguishes them is being operators on asynchronous iterators</strong>:</p> The following Stream</code> helpers / operators take a stream and perform some simple actions on it without changing the values:</p> skip_while</code>: Drops items from a stream while a condition is met (provided as a boolean closure on the items of the input stream).</li> peekable</code>: Adds a peek</code> method that can be used to preview a reference to the next item</strong> without consuming it (yet). This is useful when you don’t want to step too fast</em> through the stream.</li> take</code>: Simply skip a number of items from the beginning of the stream.</li> enumerate</code>: Adds an increasing index to the items of stream, starting from the beginning.</li> </ul> For example, you can use enumerate</code> as follows:</p> let stream = stream::iter(vec!['a', 'b', 'c']); let mut stream = stream.enumerate(); assert_eq!(stream.next().await, Some((0, 'a'))); assert_eq!(stream.next().await, Some((1, 'b'))); assert_eq!(stream.next().await, Some((2, 'c'))); assert_eq!(stream.next().await, None); </code></pre> Filtering streams</a></h3> You can filter a stream of numbers to only keep the even numbers as follows:</p> use std::future::ready; use futures::{stream, StreamExt}; let stream = stream::iter(1..=10); let events = stream.filter(|x| ready(x % 2 == 0)); </code></pre> Notice the ready</code> function. This function maps primitive Rust values into the async world</strong>. The output of ready</code> is a minimal Future</code> that can be moved: it is Unpin</code></strong>.</p> Remark</strong>: Don’t try to implement ready</code> yourself, just import it from std::future::ready</code>.</em></p> Boolean operators</a></h3> The futures</code> crate also provides analogues for the boolean operators shipped with the standard library Iterator</code> such as any</code>, all</code>, … :</p> let number_stream = stream::repeat(1).map(|n| n); let less_then_twenty = number_stream.all(|i| async move { i < 20 }); assert_eq!(less_then_twenty.await, true); </code></pre> Notice here that we don’t have to “pin” the less_then_twenty</code> stream, because Unpin</code> is not a requirement for all</code>.</p> Sinks</a></h3> Dual of streams</a></h3> Up until now we have only seen detailed usage of the Stream</code> trait. But the opposite of a stream, a “sink”, is also shipped by the futures</code> crate as the Sink</code> trait. A Sink</code> is something that receives data, agnostic from the transport</strong> or channel used.</p> The different life-cycle stages of a Sink</code> can be summarized as follows:</p> stage</th> name</th> method</th> meaning</th> remark</th></tr></thead> creation</td> new</td> </td> Initial state</td> </td></tr> send</td> ready</td> ready().await</code></td> Wait until cache ready</td> may be full</td></tr> send</td> start send</td> start_send(item)</code></td> Load into cache</td> not actual send</td></tr> send</td> flush</td> flush().await</code></td> Send items from cache</td> </td></tr> close</td> close</td> close().await</code></td> Close the Sink</code></td> not automatic</td></tr> </tbody></table> The analogue of the map function for streams StreamExt::map</code> for Sink</code>s is the sink operator SinkExt::with</code>. Instead mapping the output items of a stream, it applies a mapping function to all items that are going to be flushed into the sink.</p> If a Sink</code> becomes full easily and you depend on the concrete underlying type (such as for example a sink derived from a particular type of sender of a Tokio channel), you can allocate an extra buffer with StreamExt::buffer()</code> to cache elements that don’s fit in the sink.</p> Remark: The Sink</code> trait is not as common as the Stream</code> trait in the crates that I have used. It is, however, very easy to implement yourself.</em></p> Flushing a Stream</code> into a Sink</code></a></h3> A Sink</code> may appear in combination with a Stream</code>. In that case, it is possible to create a fully functional pipeline that takes a stream and flushes it into a sink. This is done with the forward</code> method of the Sink</code> trait.</p> let (output,_) = channel::new(); let output = PollSender::new(output); let input = stream::repeat(1).map(Ok); input.forward(output).await.unwrap(); </code></pre> Important:</p> StreamExt::forward</code> takes a TryStream</code> (items are Result</code>)</li> returns future of Result</code> (need to ignore error).</li> </ul> More Sink</code> operators</a></h3> The forward</code> method will also close the Sink</code> upon termination of the input stream. If you don’t want to close the Sink</code> after stream returned None</code>, use the sink operator SinkExt::send_all</code>.</p> When you have one input stream and know n</code> output sinks at compile-time</strong>, you can use StreamExt::fanout</code>. Otherwise you will need a mechanism to Clone</code> the input stream at run-time.</p> Splitting streams</a></h3> Collapsing an iterable of streams</a></h3> Given an iterable of streams, you can collapse the whole iterable into one stream with select_all</code>. This function will just emit the stream items from all the streams (assuming the iterator is finite) as they arrive.</p> A simple example would look like this:</p> let stream_a = stream::repeat(1); let stream_b = stream::repeat(2); let merged = stream::select_all([stream_a, stream_b]); </code></pre> In practice, you would typically pass large vectors, compile-time-sized arrays or other iterable collections to the select_all</code> function.</p>