net/http: new protocol registration mechanism

[neild]

The following is not quite a proposal. It is a declaration of an intent to commit shenanigans, and an apology (in the sense of "a reasoned argument or justification of something") for them.

The net/http package allows an external package (practically: golang.org/x/net/http2) to register an HTTP/2 server and client implementation with it. (This is Server.TLSNextProto and Transport.TLSNextProto, plus Transport.RegisterProtocol.)

The current registration mechanism is cumbersome and can't easily be extended to support some features we want to add to the package. For example, we want to add support for unencrypted HTTP/2 (#67816), but the current extension mechanism assumes all HTTP/2 connections are a *tls.Conn. We have no way to pass an unencrypted net.Conn from net/http to the HTTP/2 implementation.

We have a plan to move x/net/http2 into std (#67810), but this involves a complex sequence of steps in which adding unencrypted HTTP/2 support is supposed to occur before the package move.

Therefore, it would be very convenient in the short term to have a better connection between net/http and golang.org/x/net/http2.

Ideally, this connection would be extensible if/when we discover additional ways the two packages need to communicate. It should also add little to no exported API surface to net/http, since it will have few-to-no users.

I propose, therefore, to add the following two unexported functions to net/http, using //go:linkname to make them visible to x/net/http2 (and only x/net/http2):

//go:linkname serverRegisterProtocolImplementation golang.org/x/net/http2.nethttp_serverRegisterProtocolImplementation
func serverRegisterProtocolImplementation(s *Server, proto string, impl any) error

//go:linkname transportRegisterProtocolImplementation golang.org/x/net/http2.nethttp_transportRegisterProtocolImplementation
func transportRegisterProtocolImplementation(t *Transport, proto string, impl any) error

This is implemented in https://go.dev/cl/616097 (net/http) and https://go.dev/cl/616118 (x/net/http2).

The interface passed in the impl parameters is fiddly, low-level, contains no user-serviceable parts, and is subject to change to in the future. (We pass it as an "any" to make it easier to evolve if necessary.) See the above CLs for the details.

It is likely that we will want to expose a user-visible protocol registration mechanism in the future to support HTTP/3, since there exists at least one existing third-party HTTP/3 implementation. We could do this by converting the above unexported functions to exported methods and defining an appropriate interface for HTTP/3 server/client implementations:

func (*Server) RegisterProtocolImplementation(proto string, impl any)
func (*Transport) RegisterProtocolImplementation(proto string, impl any)

That's a separate proposal, though.

[gabyhelp]

Related Issues and Documentation

• net/http: HTTP version selection API #67814
• net/http: Export http2Transport #14394 (closed)
• net/http: support unencrypted HTTP/2 (h2c) #67816
• net/http: move HTTP/2 into std #67810
• proposal: net/http: support TLS interface for HTTP/2 #64359 (closed)
• net/http: add support for SETTINGS_ENABLE_CONNECT_PROTOCOL #53208
• x/net/http2: Exporting h2 connection setup #12737 (closed)
• proposal: x/net/websocket: implement support for websockets over HTTP/2 #53209
• proposal: net/http: allow use of external TLS stacks #21753 (closed)
• net/http: Cannot set default 'h2' and other NextProto handler concurrently #16588 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[neild]

/cc @marten-seemann because this is a step in the direction of defining a good way for HTTP/3 implementations to register themselves with net/http.