Error processing request.stream() object

[sciai-ai]

Hi,

1. I am following this code blow to stream the bytes sent in a large wav file request and then save each chunk to a wav file and do further processing on them. However eI am getting the following error

from starlette.requests import Request
from starlette.responses import Response

async def create_file(background_tasks: BackgroundTasks, request: Request):
    async for chunk in request.stream():      
        print(chunk)
         with open(wav_filename, mode='bx') as f:
            f.write(chunk)

RuntimeError: Error opening 'audio_data/ebcfefa2-f217-46ed-b52e-f49c0d0eb02e2.wav': File contains data in an unknown format.

This is the output of the print command

b'--------------------------e4462b3dae0007d5\r\nContent-Disposition: form-data; name="file"; filename="audio.wav"\r\nContent-Type: audio/wav\r\n\r\nRIFF$p\x02\x00WAVEfmt \x10\x00\x00\x00\x01\x00\x01\x00\x80>\x00\x00\x00}\x00\x00\x02\x00\x10\x00data\x00p\x02\x00\x00\x00\x00\x00\xff\xff\x00\x00\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\x00\x00\xff\xff

2. Where can i specify the number of bytes parameter for the chunks aka chunk size? The file format is wav 16Khz, 16 bits/sample. I want to have 15 seconds worth of bytes in a chunk.

[adriangb]

A couple issues I'm seeing at first glance

You're receiving multipart data

When you receive data from the client it's always a stream of bytes (think of it like a list/array where each element is a character, except binary so it's really just a number, hence the hexadecimal numbers you're seeing in your stream).

But in this case it is a multipart/form-data request. This is special encoding that allows multiple files to be sent in a single list. The basic idea is to insert a delimiter in between the files, which in your case is the --------------------------e4462b3dae0007d5 at the beginning of the file. It looks like your actual file is the form field called file. You can extract it like this:

from starlette.requests import Request
from starlette.responses import Response

async def create_file( request: Request):
    form = await request.form()
    file = form["file"]
    print(await file.read())

You are overwriting the file you create

When you do this:

async for chunk in request.stream():      
    print(chunk)
     with open(wav_filename, mode='bx') as f:
        f.write(chunk)

You are re-opening f for every chunk, so you end up only with the last chunk in the file. Instead, you'll want to do:

with open(wav_filename, mode='bx') as f:
    async for chunk in request.stream():      
        f.write(chunk)

Putting it all together

from starlette.requests import Request
from starlette.responses import Response

async def create_file( request: Request):
    form = await request.form()
    file = form["file"]
    chunk_size = 1024*1024
    try:
        with open(wav_filename, mode='bx') as f:
            chunk = await file.read(chunk_size)
            if not chunk:
                return    
            f.write(chunk)
    finally:
        await file.aclose()

[sciai-ai]

A couple issues I'm seeing at first glance

You're receiving multipart data

When you receive data from the client it's always a stream of bytes (think of it like a list/array where each element is a character, except binary so it's really just a number, hence the hexadecimal numbers you're seeing in your stream).

But in this case it is a multipart/form-data request. This is special encoding that allows multiple files to be sent in a single list. The basic idea is to insert a delimiter in between the files, which in your case is the --------------------------e4462b3dae0007d5 at the beginning of the file. It looks like your actual file is the form field called file. You can extract it like this:

from starlette.requests import Request
from starlette.responses import Response

async def create_file( request: Request):
    form = await request.form()
    file = form["file"]
    print(await file.read())

Would not await file.read() require the whole file to be stored in memory first? I want to receive bytes of chunk length and process them immediately rather than the whole file to be read first.

[adriangb]

If you really want to do this in a streaming fashion you'll have to send a non-multipart request or parse the multipart stream yourself. Starlette does not support stream processing of multipart requests.

Would not await file.read() require the whole file to be stored in memory first?

It will be loaded into memory and spill over to disk above ~1MB

[sciai-ai]

@adriangb I am fine with using a non-multipart request. Below is the curl command i am using, which part can i change to make this non-multipart?

curl -X 'POST'
'http://ip_ddress:port/get_result/'
-H 'accept: application/json'
-F 'file=@audio.wav;type=audio/wav'