Chunked download in "Monitoring download progress" corrupts large (~100MB+) files

[mjuopperi]

Using the example from here:

https://www.python-httpx.org/advanced/#monitoring-download-progress

import tempfile

import httpx
from tqdm import tqdm

with tempfile.NamedTemporaryFile() as download_file:
    url = "https://speed.hetzner.de/100MB.bin"
    with httpx.stream("GET", url) as response:
        total = int(response.headers["Content-Length"])

        with tqdm(total=total, unit_scale=True, unit_divisor=1024, unit="B") as progress:
            num_bytes_downloaded = response.num_bytes_downloaded
            for chunk in response.iter_bytes():
                download_file.write(chunk)
                progress.update(response.num_bytes_downloaded - num_bytes_downloaded)
                num_bytes_downloaded = response.num_bytes_downloaded

    print(download_file.name)
    import time

    time.sleep(1000000)

If I download the file with wget the md5 is 2f282b84e7e608d5852449ed940bfc51, with this code it is cf3e683301e09fb372dd5e3191a2dede.

File sizes also differ: 230784 for httpx and 230752 for wget.

[florimondmanca]

@mjuopperi Hi, are you able to reproduce this with other endpoints? Could it be that speed.hetzner.de simply changes the content of the /100MB.bin file (e.g. generates random content on the fly)?

I was not able to reproduce with a local setup as follows:

• Generate a random 100 MB file:

head -c $((100*1024*1024)) /dev/urandom > 100MB.bin

• Start a web server:

python -m http.server

• Get using HTTPX:

import httpx
from tqdm import tqdm

with open("download.bin", "wb") as download_file:
    url = "http://localhost:8000/100MB.bin"
    with httpx.stream("GET", url) as response:
        total = int(response.headers["Content-Length"])

        with tqdm(total=total, unit_scale=True, unit_divisor=1024, unit="B") as progress:
            num_bytes_downloaded = response.num_bytes_downloaded
            for chunk in response.iter_bytes():
                download_file.write(chunk)
                progress.update(response.num_bytes_downloaded - num_bytes_downloaded)
                num_bytes_downloaded = response.num_bytes_downloaded

    print(download_file.name)

python test.py

• Get using wget:

wget http://localhost:8000/100MB.bin

• Compare MD5's, which match:

$ md5 download.bin
36afeebff1091bae0c95e103789effd6
$ md5 100MB.bin.1
36afeebff1091bae0c95e103789effd6

• And file sizes match:

$ ls -la
-rw-r--r--    1 florimond  staff  104857600 25 Dec 19:54 100MB.bin.1
-rw-r--r--    1 florimond  staff  104857600 25 Dec 19:54 download.bin

[mjuopperi]

@florimondmanca I started investigating this because downloading this file started failing: https://www.fuzzwork.co.uk/dump/postgres-schema-latest.dmp.bz2

It's a postgres dump and I've verified it works if I download it with wget and uncompress with bzip2 cli.

When trying to uncompress this after downloading with httpx, it shows an error about the file ending unexpectedly. Could it be that the last bit of the file fails to load for some reason?

[mjuopperi]

I realized I had only been testing with smaller chunk sizes on the iterator. With 10 * 1024 chunk size the 100MB file suddenly downloaded succesfully. Any ideas what might cause this?

import tempfile

import httpx
from tqdm import tqdm

with tempfile.NamedTemporaryFile() as download_file:
    url = "https://speed.hetzner.de/100MB.bin"
    with httpx.stream("GET", url) as response:
        total = int(response.headers["Content-Length"])

        with tqdm(total=total, unit_scale=True, unit_divisor=1024, unit="B") as progress:
            num_bytes_downloaded = response.num_bytes_downloaded
            for chunk in response.iter_bytes(chunk_size=10 * 1024):
                download_file.write(chunk)
                progress.update(response.num_bytes_downloaded - num_bytes_downloaded)
                num_bytes_downloaded = response.num_bytes_downloaded

    print(download_file.name)
    import time

    time.sleep(1000000)
    ```

[mjuopperi]

I have still not been able to download the dump succesfully though. Changing the chunk size does change the downloaded file there too.

[mjuopperi]

Looking at hex dumps of the files that don't donwload properly, they are all missing stuff in the end. It looks like the last bytes are either not downloaded or not yielded by the iterator.

[tomchristie]

I've not been able to replicate this.

For example...

$ httpx https://www.fuzzwork.co.uk/dump/postgres-schema-latest.dmp.bz2 --download postgres-schema-latest.dmp.1
...
$ curl https://www.fuzzwork.co.uk/dump/postgres-schema-latest.dmp.bz2 > postgres-schema-latest.dmp.2
...
$ md5 postgres-schema-latest.dmp*
MD5 (postgres-schema-latest.dmp.1) = 2bea81ead05b699b63ec6a95608f45c0
MD5 (postgres-schema-latest.dmp.2) = 2bea81ead05b699b63ec6a95608f45c0

Also...

$ httpx https://speed.hetzner.de/100MB.bin --download 100MB.bin.1
...
$ curl http://localhost:8000/100MB.bin > 100MB.bin.2
...
$ python example.py  # Use script rather than cli to download "100MB.bin.3"
...
$ md5 100MB.bin.*
MD5 (100MB.bin.1) = 2f282b84e7e608d5852449ed940bfc51
MD5 (100MB.bin.2) = 2f282b84e7e608d5852449ed940bfc51
MD5 (100MB.bin.3) = 2f282b84e7e608d5852449ed940bfc51

[mjuopperi]

Could you share the code you're using. Or is it the one you showed previously?