Add Transfer-Encoding: chunked + Trailer

[Prinzhorn]

I'm planning on using go-httpbin for integration tests of an HTTP intercepting proxy. Thanks for maintaining it!

I will come up with some more "exotic" feature requests and have to see how easy it is to add those myself.

I'd love to see support for chunked encoding with trailers.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing

Here's a rough idea:

/chunked/:n Generates a chunked response with n random chunks of variable size, a Transfer-Encoding: chunked header and an optional Server-Timing trailer

Parameters:

• minsize=1: Min number of random bytes
• maxsize=1024: Max number of random bytes
• trailer=1: Include the Trailer: Server-Timing header and Server-Timing trailer

Does that sound like something you would accept? Do you see any problems? I don't really care about Server-Timing directly, but I need any trailer. Foo would do but I thought we might as well use something that's actually used IRL and has some browser support.

To support max-body-size we could just check the worst case maxsize * n before doing anything.

[Prinzhorn]

I've realized how little trailers are used (aside from more specific uses such as in gRPC, which runs over HTTP/2). Also some of the tooling I'm using doesn't even support them, so that part is way less important than the chunked encoding. But I think the trailers are basically free to implement.

[mccutchen]

So, at a high level, I'm in favor of some combination of a) adding a new, more explicit and hopefully discoverable /chunked endpoint b) making the Transfer-Encoding: chunked behavior of the /stream, /stream-bytes, and /drip¹ endpoints more obvious/discoverable/explicit and c) adding the ability to specify trailers to any or all of the above.

Quick demo of /stream using chunked transfer encoding:

$ curl --verbose --no-buffer --http1.1 https://httpbingo.org/stream/5 2>&1 | grep -e 'HTTP/1.1' -e 'transfer-encoding'
> GET /stream/5 HTTP/1.1
< HTTP/1.1 200 OK
< transfer-encoding: chunked

I suspect this will require a bit of fighting w/ the Golang stdlib net/http server's implementation, which automagically handles² chunked transfer encoding in some(?) cases. We get that behavior implicitly in the endpoints above via our use of the http.Flusher interface (docs, example usage in the handler for /stream).

I've never dealt with trailers myself, so I have no idea what it would take to add that support to go-httpbin. As you point out the primary use case I've seen is for gRPC, but it should at least be possible.

Footnotes

1. In poking around at this issue, I noticed that /drip does not currently send back Transfer-Encoding: chunked, at least as currently served by https://httpbingo.org/, though I expected it to. No idea if this is due to my host, or some difference in the way I'm using the Flusher interface in that handler (e.g. maybe writing a status code breaks something here). But this is kinda what I mean by having to fight w/ the stdlib to support the kind of functionality you have in mind! ↩

2. The automagical handling of Transfer-Encoding also makes it tough to write the kinds of test cases I'd like to write for these endpoints, as noted in this comment. Maybe there's a better way to do all of this, though, so any improvements here are welcome! ↩

[Prinzhorn]

Thanks for looking into this. All the things you've listed sound reasonable. I didn't know that some endpoints already support chunked.

I'll probably get to the integration tests within the next three month. From what I can tell the exiting endpoints might already get me almost 100% of the way.