Adding Bluetooth Audio output

[baesek]

Hi,
I am just trying to implement Audio Out over bluetooth using the
ESP32-A2DP Library by Phil Schatzmann https://github.com/pschatzmann/ESP32-A2DP
Here is a description of a simple sine output, which works on my Audio Kit board.
https://www.pschatzmann.ch/home/2020/09/15/sending-sound-from-an-esp32-to-a-bluetooth-sink-e-g-bluetooth-speaker/
On the bottom of the libraries github page under the headline A2DP Source (Music Sender) there is another example of how to output to a bluetooth sink.

Can someone help to implement it to the sampler? Do you think there is a way of using the aptX Live codec for bluetooth instead of the SBC to get a lower latency?

I just copied and pasted the audio_task() function and make it fit the get_data_channels callback of the bluetooth sender example but it s not working yet.

int32_t get_data_channels(Frame *frame, int32_t channel_len) {
  int32_t bufferSize = channel_len;
  /*
    static double m_time = 0.0;
    double m_amplitude = 10000.0;  // -32,768 to 32,767
    double m_deltaTime = 1.0 / 44100.0;
    double m_phase = 0.0;
    double double_Pi = PI * 2.0;
    // fill the channel data
    for (int sample = 0; sample < channel_len; ++sample) {
        double angle = double_Pi * c3_frequency * m_time + m_phase;
        frame[sample].channel1 = m_amplitude * sin(angle);
        frame[sample].channel2 = frame[sample].channel1;
        m_time += m_deltaTime;
    }
    frame[sample].channel1 = m_amplitude * sin(angle);
    frame[sample].channel2 = frame[sample].channel1;
  */

    memset(fl_sample, 0, sizeof(fl_sample));
    memset(fr_sample, 0, sizeof(fr_sample));

    Audio_Input(fl_sample, fr_sample);

    for (int n = 0; n < bufferSize; n++)
    {
        /*
         * this avoids the high peak coming over the mic input when switching to it
         */
        fl_sample[n] *= click_supp_gain;
        fr_sample[n] *= click_supp_gain;

        if (click_supp_gain < 1.0f)
        {
            click_supp_gain += 0.00001f;
        }
        else
        {
            click_supp_gain = 1.0f;
        }

        /* make it a bit quieter */
        fl_sample[n] *= 0.5f;
        fr_sample[n] *= 0.5f;

        /*
         * this removes dc from signal
         */
        fl_offset = fl_offset * 0.99 + fl_sample[n] * 0.01;
        fr_offset = fr_offset * 0.99 + fr_sample[n] * 0.01;

        fl_sample[n] -= fl_offset;
        fr_sample[n] -= fr_offset;
    }

    /*
     * main loop core
     */
    Sampler_Process(fl_sample, fr_sample, bufferSize);

#ifdef MAX_DELAY
    /*
     * process delay line
     */
    Delay_Process_Buff2(fl_sample, fr_sample, bufferSize);
#endif

    /*
     * add some mono reverb
     */
    Reverb_Process(fl_sample, bufferSize);
    memcpy(fr_sample,  fl_sample, sizeof(fr_sample));

    /*
     * apply master output gain
     */
    for (int n = 0; n < bufferSize; n++)
    {
        /* apply master_output_gain */
        fl_sample[n] *= master_output_gain;
        fr_sample[n] *= master_output_gain;

        /* output signal to vu meter */
        *vuOutL = max(*vuOutL, absf(fl_sample[n]));
        *vuOutR = max(*vuOutR, absf(fr_sample[n]));
    }

    //Audio_Output(fl_sample, fr_sample);
    frame[n].channel1 = fl_sample;
    frame[n].channel2 = fr_sample;
    
    Status_Process_Sample(bufferSize);
    
    return channel_len;
}

best,
paul

---

EDIT: I am trying to understand how to the audio is played back
this is the call from the main loop. Is the main loop called at 240Mhz?

void loop()
{
    audio_task(); /* audio tasks blocks for one sample -> 1/44100s */

What does the comment mean? As I understand the audio_task() function processes the two buffers of 64 32bit values for left and right and then it sends alle 64 values to the dac.
Here i put some comments and questions into the functions code for better understanding:

static float fl_sample[SAMPLE_BUFFER_SIZE];
static float fr_sample[SAMPLE_BUFFER_SIZE];

#ifndef absf
#define absf(a) ((a>=0.0f)?(a):(-a))
#endif

/*
 * the main audio task
 */
inline void audio_task()
{
    memset(fl_sample, 0, sizeof(fl_sample));
    memset(fr_sample, 0, sizeof(fr_sample));
    // Note for understanding: 
    // This sets all values in the static array of 64 (thats the default SAMPLE_BUFFER_SIZE) values of 32bit float type to zero.
    // In other words there are two sample buffers which store 32bit values and 64 values of storage that get resetted to zero.

    Audio_Input(fl_sample, fr_sample);
    // Note for understanding:
    // This reads 64 32bit values to the two buffer arrays from the Analog to digital Converter via the I2S interface
    // What if there is no audio input? or if it is deactiated? 

    for (int n = 0; n < SAMPLE_BUFFER_SIZE; n++)
    {
        /*
         * this avoids the high peak coming over the mic input when switching to it
         */
        fl_sample[n] *= click_supp_gain;
        fr_sample[n] *= click_supp_gain;

        if (click_supp_gain < 1.0f)
        {
            click_supp_gain += 0.00001f;
        }
        else
        {
            click_supp_gain = 1.0f;
        }

        /* make it a bit quieter */
        fl_sample[n] *= 0.5f;
        fr_sample[n] *= 0.5f;
        

        /*
         * this removes dc from signal
         */
        fl_offset = fl_offset * 0.99 + fl_sample[n] * 0.01;
        fr_offset = fr_offset * 0.99 + fr_sample[n] * 0.01;
        
        fl_sample[n] -= fl_offset;
        fr_sample[n] -= fr_offset;

        /*
         * put vu values to vu meters
         */
        *vuInL = max(*vuInL, absf(fl_sample[n]));
        *vuInR = max(*vuInR, absf(fr_sample[n]));
    }
    // Note for understanding:
    // This whole iteration only affects 64 32bit samples of Audio input data so far

    /*
     * main loop core
     */
    Sampler_Process(fl_sample, fr_sample, SAMPLE_BUFFER_SIZE);
    // Note for understanding:
    // This function overrides each of the two buffers with the next 64 32bit values the sampler needs to output

#ifdef MAX_DELAY
    /*
     * process delay line
     */
    Delay_Process_Buff2(fl_sample, fr_sample, SAMPLE_BUFFER_SIZE);
    // Note for understanding:
    // I dont know what this function does as i cannot find its implementation
#endif

    /*
     * add some mono reverb
     */
    Reverb_Process(fl_sample, SAMPLE_BUFFER_SIZE);
    memcpy(fr_sample,  fl_sample, sizeof(fr_sample));
    // Note for understanding:
    // I think that overrides the right buffer with the left one and thus kills all stereo.

    /*
     * apply master output gain
     */
    for (int n = 0; n < SAMPLE_BUFFER_SIZE; n++)
    {
        /* apply master_output_gain */
        fl_sample[n] *= master_output_gain;
        fr_sample[n] *= master_output_gain;

        /* output signal to vu meter */
        *vuOutL = max(*vuOutL, absf(fl_sample[n]));
        *vuOutR = max(*vuOutR, absf(fr_sample[n]));
    }
    // Note for understanding:
    // This just multiplies all values of the buffer with a master_output_gain value.
    // I dont know what the range of the 32bit sample values is. Is it 3.4028235E+38 to +3.4028235E+?

    Audio_Output(fl_sample, fr_sample);
    // Note for understanding:
    // That prepares the two buffers to be transmitted to the DAC using the I2S Interface.


#ifdef OLED_OSC_DISP_ENABLED
    ScopeOled_AddSamples(fl_sample, fr_sample, SAMPLE_BUFFER_SIZE);
#endif

    Status_Process_Sample(SAMPLE_BUFFER_SIZE);
}

[baesek]

Now i got the first output via bluetooth using the following, but it seems that it is not okay to just override channel_len. The sound is just stuttering...

void loop()
{
#ifndef BLUETOOTH_ENABLED
    audio_task(); /* audio tasks blocks for one sample -> 1/44100s */
#endif
...
}

#ifdef BLUETOOTH_ENABLED 
int16_t data_arrays[2];
// callback used by A2DP to provide the sound data
int32_t get_data_channels(Frame *frame, int32_t channel_len) {
    audio_task();
    channel_len = SAMPLE_BUFFER_SIZE;
    // update the frame with converted buffer values to int16_t 
    for (int sample = 0; sample < channel_len; ++sample) {        
        frame[sample].channel1 = int16_t(fl_sample[sample] * 16383.0f);
        frame[sample].channel2 = int16_t(fr_sample[sample] * 16383.0f);
    }
    return channel_len;
}
#endif

[baesek]

I now added a FIFO using the CircularBuffer Library from Roberto Lo Giacco.

// Bluetooth output
#include <CircularBuffer.h>
#include "BluetoothA2DPSource.h"
BluetoothA2DPSource a2dp_source;
...
static CircularBuffer<int16_t,SAMPLE_BUFFER_SIZE * 8> fl_sample_bt;
static CircularBuffer<int16_t,SAMPLE_BUFFER_SIZE * 8> fr_sample_bt;

#ifdef BLUETOOTH_ENABLED 
// callback used by A2DP to provide the sound data
int32_t get_data_channels(Frame *frame, int32_t channel_len) {
    //audio_task();
    // channel_len = fl_sample_bt.size();
    for (int sample = 0; sample < channel_len; ++sample) {
        // update the frame with converted buffer values to int16_t 
        //frame[sample].channel1 = int16_t(fl_sample[sample] * 16383.0f);
        //rame[sample].channel2 = int16_t(fr_sample[sample] * 16383.0f);
        frame[sample].channel1 = fl_sample_bt.pop();
        frame[sample].channel2 = fr_sample_bt.pop();
    }
    return channel_len;
}
#endif
...
void setup()
{
    // put your setup code here, to run once:
    delay(500);
#ifdef BLUETOOTH_ENABLED    
    a2dp_source.start("FeinTech-ABT00101", get_data_channels);  
    a2dp_source.set_volume(100);
#endif
...

void loop() 
{
audio_task();
...
}
...
inline void audio_task()
{
    ...
    /*
     * apply master output gain
     */
    for (int n = 0; n < SAMPLE_BUFFER_SIZE; n++)
    {
        /* apply master_output_gain */
        fl_sample[n] *= master_output_gain;
        fr_sample[n] *= master_output_gain;
#ifdef BLUETOOTH_ENABLED
        fl_sample_bt.unshift(int16_t(fl_sample[n] * 16383.0f));
        fr_sample_bt.unshift(int16_t(fr_sample[n] * 16383.0f));
#endif 
        /* output signal to vu meter */
        *vuOutL = max(*vuOutL, absf(fl_sample[n]));
        *vuOutR = max(*vuOutR, absf(fr_sample[n]));
    }
    ...
    Audio_Output(fl_sample, fr_sample);
    ...

It works somehow, but the sound is kind of distorted...

EDIT:
If I set the FIFO size to much bigger:

static CircularBuffer<int16_t,SAMPLE_BUFFER_SIZE * 32> fl_sample_bt;
static CircularBuffer<int16_t,SAMPLE_BUFFER_SIZE * 32> fr_sample_bt;

It is also much cleaner, but still sometimes there are some noises and clicks.
It seems like the clicking is happening in an interfering pattern, in the beginning there is no clicks, then they appear, get stronger, and then disappear again.

EDIT2: I am not sure where this crackling comes from. Could it be that Interrupts are the problem?

Now i have much noise and crackling in the beginning (i think one fifo size)

[marcel-licence]

Sounds good. You could try to record the audio and check the distance of the clicks in samples. That might provide some information about the origin of it.

[baesek]

Okay I asked the author of that library. He explained that the get_data_channels is called from the Espressif IDF and is suited best for a polling API. In my case the requested channel_ len is always 128, while the Buffersize is set to 64. He assumed that the problem could be a buffer underrun or a synchronization problem with the two cores.

Is Esp32_midi_sampler using both cores? In my Arduino settings it says Events on: Core 1, Arduino on Core 1...

I am not sure how easy the code can be transformed to poll the 128 * 2 16 bit values for the callback...

Sadly APTX low latency codec is not supported yet by Espressif, so there will be latency of around 200ms when using the BT SBC Codec)

[marcel-licence]

The audio part of the sampler uses only one core. The second core is used for non time critical operations. If I would find a little Audio Input / Output example I could give it a try.
The audio part does reads the input audio buffer. The data will be processed and finally put into the output buffer. It looks like that the I2S transfer runs in the background (DMA?!). You could change the size of the audio buffer used by the sampler to match the bluetooth audio buffer.
How is the bluetooth audio synced? How can you ensure that you do not write samples into the frame buffer during read/write operations of the bluetooth lib?

[marcel-licence]

I tried out the BT audio connection as well. It does work. I was hoping to get both directions working like a BT headset but it seems it refuses to use the sink and source at the same time.
I could see that the callback provides 2560 bytes of data send from a windows computer. It seems that the connection is not stable enough and needs a bigger internal buffer to avoid drop outs. In my case I was just using the sink to push audio through the DIY effects on the audio kit.
In addition to that it looks like the sink requires a lot of heap. I am not sure if that would fit in the synthesizer projects.