Playback API

[JayPanoz]

We will implement Playback to Readium Speech in the upcoming weeks. The Playback API should make things consistent across Readium Projects and help streamline implementations and adapters to remote services.

Here's something we can start iterating over, with the requirements we discussed during a call.

https://gist.github.com/JayPanoz/24dbad1bbf284124d43394ef66b2cbb5

[HadrienGardeur]

Just to make sure that everyone is aware of this discussion cc @mickael-menu @qnga @chocolatkey @danielweck @panaC @GoobyTheBOI

[JayPanoz]

The gist has been revised following feedback and inputs we could get since Monday. 🙏

[danielweck]

From the conference call, a few thoughts based on our experience with Thorium's TTS readaloud feature:

• Continuous vs. discrete (automatic vs. manual) progression from one utterance to the next. Note that in Thorium, the user preference (GUI toggle switch) also applies to playback of pre-recorded audio clips in EPUB3 Media Overlays / DAISY digital talking books. Also note that there is currently no setting to automatically pause at the end of HTML documents / fixed-layout pages, but to automatically read the entire document uninterrupted. Also note that Thorium enters the "paused" state when awaiting user interaction, as otherwise the "stopped" state wouldn't be able to resume TTS (e.g. when user invokes the "resume" command, or "previous"/"next", or simply clicks on arbitrary text position in the document).
• Skippability enable/disable configuration (e.g. optional / ancillary speech content, such as page breaks identified at authoring time via semantic role in HTML markup). Note: this could be handled upstream in the utterances builder: statically-generated "speech content" could simply exclude skipped items, and the data structure would be invalidated / rebuilt when the user toggles the switch.
• Sentence detection / text fragmentation on-off toggle. Note: this could also be handled upstream when constructing speech content. Thorium implementation detail (not exposed to app / API surface): there are sub-indices for speech utterances constructed by the sentence splitter / text fragmenter. Internally, this distinction is useful for dealing with DOM structural constraints such as when there is a "span" inside a "paragraph" element which otherwise only contains text (i.e. pure text node before and after the "span", in the same sibling hierarchical plane relative to the parent "paragraph"), and the "span" has a different language than the "paragraph" => because we do not use SSML, this DOM structure imposes the isolation of the "span" as a distinct atomic utterance, and the construction of left/right speech utterances for the surrounding text nodes which are spoken in a different language. However all 3 utterances can be broken down further into "sentences" (text fragmentation with punctuation), thus the additional layer of indices within the primary structurally-imposed utterance.
• Word boundary events and resulting sub-utterance highlighting: in Thorium the IPC (Inter Process Comms) layer introduces a perceivable delay when dealing with events between webviews, so we avoided abstracting TTS speech and highlighting mechanics ("decorators") too far away in the component architecture. In other words, we do not rely on event propagation to trigger highlighting as a response to TTS play/pause/stop, instead we directly invoke drawing logic (invalidation of the previous word, highlight of the new one). This ensures instant visual feedback during TTS, even when using complex polygonal highlighting styles such as line masking. Note that in Thorium there is in fact a well-known performance issue when EPUB3 Media Overlays (authored SMIL synchronization) is granular at the word level as opposed to typical sentence-level or even paragraph-level (the audio/visual lag is especially perceivable with large documents) ... that is because we rely on the slow event bus to synchronize the audio-visual state.