New streaming model

[toots]

This PR implements the long-await change of our content-generation API.

With these changes, frames, our internal notion of content chunks being prepared for output, become variable-length small collections of tracks.

This should finally moves us away from the fantastic footgun that our current breaks/partial fill for track marks was.

Notes

Because track marks could appear before as part of the runtime streaming cycles, they were often implicit. For instance, if a source failed, a track mark would implicitly be added.

With the new API, we have to manually specify each track marks. This leads to situations were we don't really have a convention for. Typically, should all sources emit an initial track when starting?

If we want to be true to our muxers, it should be possible to drop all track marks, thus the answer to the above question is no. However, we have relied in the past on operators such as on_track to detect when a source becomes available again. this would also require to change existing code and, most likely, introduce a new operator.

Most of the current work was focused on stabilizing our current set of tests. Next, we shall cleanup and implement the new conventions and operators that we need.

Changed operators

The following operators are directly impacted by these changes:

• crossfade: the non-conservative mode has been removed. It was already complicated enough to keep the conservative mode. Other than that, the operator is expected to operate as close as possible to the the existing one.
• source.dynamic, switch and sequence: these operators , the only internal ones which deal with dynamic selection of source within a streaming cycle, have been rewritten to use a factored-out Source.generate_from_multiple_sources class. This class is charged with capturing all the logic pertaining to composing a full frame from multiple sources. Although tricky to finalize, it seems to be satisfying. Most of the tricks were about:
  • Triggering on_track handlers at the right time when switching sources to allow implemtations based on them such as rotate, random etc to update their internal state accordingly.
  • Knowing when to allow the operator to select a new source or keep the existing one based on partial fill and knowing when to consider a track mark a freshly selected source new track or an already selected source end of track, triggering the re-selection of a new track.

New API

The API for generating frames becomes, for sources:

       (** Sources must implement this method. It should return [true] when
           the source can produce data during the current streaming cycle. *)
       method virtual private can_generate_frame : bool

       (** Sources mushc implement this method. It should return the data
           produced during the current streaming cycle. Sources are responsible
           for producing as much data as possible, up-to the frame size setting. *)
       method virtual private generate_frame : Frame.t

       (** This method is based on [can_generate_frame] and has the same value through
           the whole streaming cycle. *)
       method is_ready : bool

       (** If the source is ready, this method computes the frame generated by the
           source during the current streaming cycle. Returned value is cached and should
           be the same throughout the whole streaming cycle. *)
       method get_frame : Frame.t

       (** This method passes the frame returned by [#get_frame] to the given callback.
           The callback should return the portion of the frame (of the form: [start, end))
           that was effectively used. This method is used when a consumer of the source's data
           only uses an initial chunk of the frame. In this case, the remaining data is cached
           whenever possible and returned during the next streaming cycle. Final returned value
           is the same as the partial chunk returned for the callback for easy method call chaining. *)
       method get_partial_frame : (Frame.t -> Frame.t) -> Frame.t

       (** This method requests a specific field of the frame that can be mutated. It used used
           by a consumer of the source that will modify the source's data (e.g. [amplify]). The
           source will do its best to minimize data copy according to the streaming context. Typically,
           if there is only one consumer of the source's data, it should be safe to pass its
           data without copying it. *)
       method get_mutable_content : Frame.field -> Content.data

       (** This method is the same as [#get_mutable_content] but returns a full frame with the request
           mutable field included. *)
       method get_mutable_frame : Frame.field -> Frame.t

       (** By convention, frames produced during the streaming cycle can only have at most one
           track mark. In case of multiple track marks (which most likely indicate a programming
           problem), all subsequent track marks past the first one are dropped.

           This function returns a pair: [(initial_frame, new_track option)] of an initial frame
           and, if a track mark is present in the frame, the optional portion of the frame contained
           after this track mark.

           This method is pretty convenient to implement operations that should be aware of new tracks. *)
       method private split_frame : Frame.t -> Frame.t * Frame.t option

TODO

• Proof read
• Test operators
• Update doc

Future work

• We should enforce that sources can produce at least some data when they return can_generate_frame. This needs some adjustments with synchronous operators such as input.ffmpeg
• Frames do not have to contain at least one video frame. We should be able to support frames of any arbitrary length.
• Reimplement clocks. Currently, get_mutated_field is not optimized yet. This requires comme coordination with the clock (e.g. to know what operators are consuming a given source's data during the current streaming cycle) that should be implemented when reworking the clocks.

[toots]

Ok, I've run a comparison test between main and this PR with the complex script that generates this VJ stream:

Memory and CPU profiles are pretty similar with a little saving on the new streaming side it seems. Overall, this is ready for merge. We might be able to save more memory allocations once we allow the frame to go down below one single video frame. Going from 4069 audio samples to 1024 could divide by 4 the amount of memory allocated in source frames!

main branch

CPU

Memory allocations

This value always feels not very reliable:

This seems more accurate:

new-streaming branch

CPU

Memory allocations