Incomplete message body received when streaming response

[kpfleming]

(This appears to be related to #201, where this exception was added)

I've got some code (below) running in Python 3.8.3 (as packaged in RHEL 8), using httpx 0.17.1. I'm seeing random cases where an exception is thrown claiming that the connection was closed before the complete message body was sent, but I suspect the actual situation is that the connection was closed by the server after it had sent the entire body but before the entire body had been read from the response object.

Code:

async def blob_getter(client: httpx.AsyncClient, blob_dir: pathlib.Path, blob: BlobDetails) -> None:
    blob_path = blob_dir / blob.digest
    if blob_path.exists():
        logger.debug("%s blob %s already present", blob.purpose, blob.name)
        return
    logger.debug("Getting %s blob %s", blob.purpose, blob.name)
    try:
        blob_url = blob.job.name + "/blobs/" + blob.digest
        async with client.stream("GET", blob_url) as response:
            if response.is_error:
                logger.error(
                    "Failed %s blob %s - %s - %s",
                    blob.purpose,
                    blob.name,
                    response.status_code,
                    response.reason_phrase,
                )
                return
            async with aiofile.async_open(blob_path, "wb") as blob_file:
                async for chunk in response.aiter_bytes():
                    await blob_file.write(chunk)
        logger.debug("Completed %s blob %s", blob.purpose, blob.name)
    except asyncio.CancelledError:
        blob_path.unlink(missing_ok=True)
        logger.debug("Cancelled %s blob %s", blob.purpose, blob.name)
    except Exception as exc:
        blob_path.unlink(missing_ok=True)
        logger.error("Failed %s blob %s - %s", blob.purpose, blob.name, str(exc))
        raise

I'm seeing exceptions caught by the except Exception clause, and I'm 100% positive that the server did send the full body, but the socket-closure was noticed before the response had been fully ready.

For now, I'm going to catch this particular exception and 'absorb' it so that I can continue reading, but I'm wondering if there's something else I should be doing.

[tomchristie]

Would you be able to reduce this down to an easily reproducible stand-alone example?

[kpfleming]

I think so, although reproducibility depends on the behavior of the server sending the content, so I'm not sure how reliable it will be. I'll give it a go.

[kpfleming]

Well, it appears that my assumption was incorrect :-) If I 'absorb' the httpx.RemoteProtocolError exception and continue reading, the resulting file contents are incorrect (a calculated digest does not match the expected value). If I retry the operation a few times, it will eventually succeed, so it's not clear to me what is happening here.

[kpfleming]

I have now confirmed that this problem is being caused by something in our network; when I bypass that 'something', I don't see this occur at all, and all of my parallel GET operations complete successfully. This discussion can be closed :-)