AudYoFlo: Use Cases: File Reader and Writer Chains
File reader functionality is required to play audio (or video) data which is available in a pre-recorded location. A very simple case is a file on the harddrive containing audio samples. Considering this in a more general sense, however, web streams are also a specific type of a pre-recorded file where the data is taken from the sink a web connection. This could be an icecast endpoint as well as an MPEG-TS multicast endpoint - what all file reader applications have in common is that there is no direct relation between input and output and it is considered as a uni-directional data flow. As a consequence, this implies that a certain delay can be tolerated which enables the developer to add a good portion of buffering to prevent dropouts in case data is stuck somewhere.
The same also applies for file writer functionality where data can be temporarily stored in a buffer to better deal with cases where the data is stuck somewhere in the output processing chain. For example, most streaming standards such as icecast or HLS typically are based on the recording of small fragments of recorded media which after afterwards uploaded to a content delivery network (CDN).
In this chapter, at first, the functionality of the file reader shall be described. Special focus will be put on threading and buffering issues. The first part will be followed by a description of the filewriter functionality which is somewhat similar to the reader in reverse order of processing.
The typical scenario of a generalized file reader is shown in the following diagram.
The input data is pre-recorded in a file and stored on the harddrive. The required steps are pretty obvious: the data must be read from the file by the Reader, then converted to a representation that is used in all components in the Converter and finally further processed - either immediately played via loudspeaker or mixed to a signal which is forwarded to other data flow processor components.
The reading of the file, however, may impose some problems: if. e.g., the data storage medium is a USB stick, the read process may take a while. This, unfortunately, may cause a temporary gap in the continuous flow of data samples which then will be audible as a dropout.
The problem related to the dropout may have two different causes:
- The data rate to read audio samples from the file may be unsufficient. This cause typically does not really occur. If it would we would need to store the input file at a different location.
- Even though the data rate is sufficient, a short moment of time may required to access the USB stick from time to time. If this moment happens at the wrong time, the dropout will occur.
Typically, cause 2) can be observed even on systems with a high speed harddrive, e.g., if the user runs another process with a lot of file i/o in parallel.
In order to prevent this kind of dropouts, a temporary buffer can be employed to store data in advance. For example, a buffer may be in use to store 1 second of audio data. If the file reader blocks for a certain moment of time, the data can be taken from the buffer which tends to be unfilled for the single moment. However, the blocking moment is much shorter than the 1 second of audio data within the buffer so that the dropout will not be audible as long as there is enough data in the buffer to realize a continuous flow of data. And even though the buffer fill height will drop for the short moment of the blocked read, it can be re-filled later by reading data with a rate higher than the rate of the draining audio samples towards the audio output.
In the following figure, a buffer is introduced right after the converter to store audio data temporarily.
In the diagram, two different threads are shown which are operated in parallel. In case the file read may be blocked by the file i/o, we need to make sure that data is continuously proveded for the audio output. This is shown on the right side and denoted as thread #2. In paralellel, the data read process is realized in thread #1. If thread #1 blocks for a while since the file reading peripherie can not deliver, thread #2 continues to output the data from the buffer. And then, if thread #2 is no longer blocked, the buffer refill may be operated at a rate faster than the rate at which data is consumed by thread #2.
In AudYoFlo, the file reader is typically driven by the output (shown as the speaker) or a mixing functionality that is driven by a soundcard. Thread #2 therefore typically is started and administred somewhere outside the file reader involved components.
In the previous figures, there was always a block denoted as the Converter. This block is required to prepare the data to be presented in the format as expected by the following components. Typically, audio data is processed while being available in data format jvxData. From the file, typically other format representations are read:
- Uncompressed file sources such as WAV files typically contain PCM samples in fixed point representations involving 16,24, 32 or even 64 bit resolution. The converter must convert the fixed point samples into a representation in
jvxDataformat. Also, often, the data is stored in interleaved format whereas in AudYoFlo, typically, non-interleaved audio sample groupings are used to better separate audio channels. - Compressed file sources such as MP3 or M4A files store audio data in short byte buffers. When producing the files, the bytes are produced by encoding the data. The other way around, a decoding is required to reconstruct audio samples from the compressed byte buffers. The converter in this case involves a decoder followed by a type conversion as described at 1).
- If the samplerate of the input file diverges from that expected by the output, the audio sampes must be up- or downsampled. In this case, the converter may also involve a resampler.
- If the number of channels does not fit to the output, a channel mapper must be part of the converter module.
In AudYoFlo, the described chain of components can well be organized by the components as shown in the following figure:
- The Reader is realized by a device defined in module
jvxAuTFileReader. That module provides a technology that exposes multiple devices dynamically which are activated by providing a filename to the technology. - The Converter is realized by an audio node of type
jvxAuNBitstreamDecoder. That node receives the byte buffers from the Reader and involves an audio decoder to convert the bytes into audio samples in buffers. - The chain ends with a component from module
jvxAuNForwardBufferwhich realizes the Buffer inbetween the reader and the rest of the processing functionality in the application.
The reader technology has a property `` which accepts a string that contains the name of an input file. Once given to the technology, it tries to open the file and determines the audio properties. If successful, the reader technology adds a reader device. The new device is reported to the host by emitting a host command report message object:
CjvxReportCommandRequest theRequest(
jvxReportCommandRequest::JVX_REPORT_COMMAND_REQUEST_UPDATE_AVAILABLE_COMPONENT_LIST,
this->_common_set.theComponentType);
The report of the new device will end up in the automation component of this project. In that component, a data connection can be established by involving a converter and a forward buffer.
During processing, the reader expects the chain to pull data from the file. It therefore declares the connection of data flow type jvxDataflow::JVX_DATAFLOW_PUSH_ON_PULL. The reader is triggered on a non-regular time basis as the element further down the processing chain controls the reader to deliver new data when the intermediate buffer requires new data.
A simple file reader for wav files is realized in module jvxAuTFileReader. It only reads WAV files but understands all kinds of WAV files for 32 bit as well as 64 bit addressing. The reader takes the specificiations of the file to derive samplerate, number of channels as well as the sample format from the opened file. The buffersize is not fixed. Nevertheless, a buffersize of 1024 is setup by default and can be adapted when the device is active. Devices can be produced by adding the filename as a string via property open_filename. If the file exists it will be associated to the device which should be autoconnected to a Converter and a Forward Buffer.
In the processing chain, the module forwards buffers of type JVX_DATAFORMAT_BYTE with format group JVX_DATAFORMAT_GROUP_AUDIO_CODED_GENERIC. Since the format of the transmitted byte field is not described in the bitstream a format_spec token is passed from component to component. This file reader can be identified by the family assignment in the format specification, e.g.,
format_spec = fam=jvx;tp=wav32;ch=2;sr=48000;bs=1024;fsm=4096;pcmfi;le;bps=16
The entries in the format specification are as follows:
-
fam=jvx: Coded data requires a decoder of family typejvx, -
tp=wav32: Coded data is taken from a 32 bit WAV file, -
ch=2: Coded data includes 2 channels of audio data, -
sr=48000: Coded data is available at a samplerate of 48 kHz, - 'bs=1024': Each frame of coded data is decoded to 1024 sampes,
-
fsm=4094: Maximum number of bytes in a packet, -
pcmfi: PCM data is stored in fixed point arithmetic, -
le: Data in little endian format, -
bps=16: Number of bits per sample is 16.
The format specification is at first used by the following converter to find the right decoder / decoder family. Then, the token is passed to the decoder to configure it properly.
This file reader uses the FFMPEG library to read data from the specified file. It supports all typical file formats and extends the reader functionality towards mp3 and m4a (AAC) formats. The data format that is passed through the processing chain, however, is out of the scope of the typical AudYoFlo concept since every frame is associated to a specific buffer when being read from file which is then transferred from component to component. The format group is therefore of type JVX_DATAFORMAT_GROUP_FFMPEG_BUFFER_FWD to indicate that a dedicated format is in use.
The converter of type jvxAuNBitstreamDecoder is a proxy to hold the appropriate decoder for the byte buffers as received from the reader. Within the connection towards the next components, the filereader adds a config token in the jvxLinkDataDescriptor, sub field jvxLinkDataDescriptor_con_params, variable format_descriptor. This config token is used by the converter to select a codec family and the decoder. Then, if the correct decoder is available, it will be integrated into the processing chain by using a fixed connection.
The buffer, or forward buffer, collects the data obtained from the reader and decoded by the decoder. It uses internally a linear multi channel audio sample buffer that acts as a FIFO. On the side towards the file reader, data is obtained from the file source whereas on the side towards the output, the component provides data. The data is obtained by the following connected component on the basis of the jvxDataflow::JVX_DATAFLOW_PUSH_ON_PULL principle to request data.
The buffer typically decouples the following two threads
- The thread to obtain data from the source in the chain where the forward buffer is located. This thread is owned by the buffer and drives all components to the left (direction: towards source).
- The thread where data is requested from the forward buffer. This might be driven by the output processing thread or a similar task. Often, the thread is driven by any kind of mixer where all data is brought together. This part is on the right side of the forward buffer - towards the sink.
This buffer is denoted as forward buffer: in this buffer, the signal delay from input to output does not play any important role. This is specifically given for input only and output only sources and sinks.
The forward buffer can be used on the input side (as it is the case in this example) as well as on the output side (example of file writer). The implementation must be switched to be used as input or output buffer by means of the property /buffer_location.
Besides the functionality to hold samples in a buffer the forward buffer may have different buffersize constraints for input and output. Also, a simple dynamic channel mapper is available via property /channel_selection.