Skip to content

Transport Protocol

Jump to bottom
Harry Zhang edited this page Jun 18, 2018 · 7 revisions

Overview

The Transport Protocol is divided into the following layers:

  1. The Connection Layer: The layer that establishes a two-way, kept-alive, stateless connection between the Client and the Server, and allows FRAMES to be transported.
  2. The Section Layer: The layer that allows the performant, ordered, safe transportation of SECTIONS.
  3. The Data Layer: The layer that allows the transportation of JSONs. Heartbeat happens here.

The Transport Protocol demands that applications provide an Handshake JSON to send directly after the connection is opened. This JSON must be provided when opening the connection.

Interface

The Interface of the Client-side Connection.

Note that this is not the documentation of the actual implementation in bilibili-danmaku-client. This section only describes the exposed interface of the Transport Protocol to Clients and Servers in pseudocode. Also, the implementation might not 100% conform to this interface. For the implementation documentation, see the code.

open()

    function open(options: ServerConnectionOptions | ClientConnectionOptions): Connection

Open a new Connection.

  • param options: The options used to open the connection. Required.
  • return: The Connection construted. Non-null.

ConnectionOptions

    class ConnectionOptions {
        url: String,
        handshakeJson: Object,
    }
  • property url: The URL representing the Server to connect to. Required.
  • property handShakeJson: The Handshake JSON to send to the Server. Required.

Connection

Methods are listed seperately.

    class Connection {
        url: String,
        state: String,
    }
  • property url: The URL representing the Server that this Connection connects to. Read-only.
  • property state: The string representing the state of this Connection. Read-only. See lifecycle.

Connection.close()

    function Connection.close(): void

If Connection is in Opening State or Opened State, switch state to closing and request Connection to close. The Connection will only be closed after the close event. See lifecycle.

Connection.send()

    function Connection.send(data: Object): void

If Connection is in Opened State, send given data to the other side of this Connection.

  • param data: The data to send. Required.

Connection.on()

    function Connection.on(event: String, callback: Function): void

Add callback as listener to event. Recommended to call this right after open(). Listeners will be called on order of on(), if the former listener throws an error, behavior is undefined. For event definitions see events.

Lifecycle

The Transport Protocol defines the following states:

  1. Opening State: When the connection is opening, when any of the 3 layers haven't finished setting up.
  2. Opened State: When the connection is established and the Data Layer is ready to send data.
  3. Closing State: When the connection is closing, after Connection.close() is called and closing is reqested.
  4. Closed State: When the connection is closed.

The states transfers as follows:

  • Start from Opening State: Right after open().
  • From Opening State
    • to Closing State: When Connection.close() is called before the connection is opened.
    • to Closed State: When an error occurs while opening. Emit event close. Emit event error with the error thrown.
    • to Opened State: When the connection is successfully established. Emit event open.
  • From Opened State
    • to Closing State: When Connection.close() is called.
    • to Closed State: When any of the layers closed or thrown an error automatically. Emit event close. If an error was thrown, emit event error with the error thrown.
  • From Closing State:
    • to Closed State: When connection is successfully closed. Emit event close.
  • End in Closed State.

Connection Layer

The Connection Layer is rather simple. Currently, the Transport Procotol simply uses a WebSocket under scema wss: and transports FRAMES by transporting their binary data with binary data frames.

On receiving a data frame, if it is a binary data frame, the frame is passed on to the Section Layer. Otherwise, discard it.

In Opening State, the Connection Layer initializes first and tries to establish a WebSocket connection using Connection.url. If an error occurs, the Connection Layer fails to initialize.

In Opened State, the Connection Layer accepts errors and close requests from the Server, according to the WebSocket Protocol. On errors the Connection Layer throws the error. On Server-requested close, the Connection Layer closes.

In Closing State, the Connection Layer closes the underlying WebSocket using the 1006 Going Away code and reason. The Connection Layer closes after the WebSocket is closed.

Section Layer

The Section Layer introduces FRAME and SECTION.

A FRAME is an array of SECTIONs. The binary data of a FRAME is the concatenation of the binary data of its SECTIONs. Here concatenation means appending the bytes of the operands in encounter order.

A SECTION has 2 parts: a HEADER part and a DATA part. The binary data of a SECTION is the concatenation of the binary data of the HEADER part and the DATA part.

The HEADER is has 4 properties:

  • length: int: The length of the whole SECTION, the HEADER included.
  • protoVer: int: The version of the current Transport Protocol. Currently 2.
  • controlFlag: boolean: Whether this SECTION is used for controlling and not data transporting.
  • opCode: int: The operation code of this SECTION.
  • binaryFlag: boolean: Whether this SECTION contain non-JSON data.

Its binary data, with 16 bytes is represented as:

    0                             1
    0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    LL LL LL LL VV VV 00 0C PP PP PP PP 00 00 00 0B

Where:

  • L means length, encoded as 32-bit Big-Endian unsigned integer.
  • V means protoVer, encoded as 16-bit Big-Endian signed integer.
  • C means controlFlag, encoded 1 for true and 0 for false.
  • P means opCode, encoded as 32-bit Big-Endian signed integer.
  • B means binaryFlag, encoded 1 for true and 0 for false.

The DATA part is a arbitrary array of bytes, named data. Its binary data is exactly its byte array.

In Opening State and Closing State, the Section Layer does nothing special.

Bundling and debundling

If the Data Layer want's to transfer several SECTIONs in a short period of time, the Section Layer can choose to bundle these SECTIONs together into one single FRAME to take advantage of system resources. When and how do bundling take place is for the implementation to decide.

On receiving a FRAME from the Connection Layer, it should be debundled into one or more SECTIONs, which MUST be implemented. The debundling process happens as follows:

  1. Let offset be 0.
  2. Try to read 16 bytes from offset and form a HEADER part, call it header. On error, throw it.
  3. Try to read header.length - 16 bytes from offset + 16 and form a DATA part, call it data. On error, throw it.
  4. Form a SECTION with header and data, pass the SECTION to the Data Layer.
  5. Set offset to offset + header.length. Jump to step 1.

If the debundling process throws an error, the Section Layer throws the error.

Data Layer

This section only describes the Data Layer on the client side.

The Data Layer introduces 5 types of SECTIONs.

controlFlag opCode binaryFlag data
Handshake Section true 7 false custom
Handshake ACK Section true 8 true []
Data Section false 5 false custom
Heartbeat Section true 2 true '[object Object]'
Heartbeat ACK Section true 3 true [0x00 * 4]

And defines 3 processes: the Handshake Process, the Convertion Process, and the Heartbeat Process. All processes run on the Data Layer.

Handshake Process

In the Handshake Process, the Client makes a handshake with the Server and establish the connection.

In Opening State, a Handshake Section is sent with the handshakeJson param passed to open(). If no Handshake ACK Section is received after a period of time, the Data Layer fails with an error. Otherwise, it finished opening.

Convertion Process

On receiving a SECTION from the Section Layer, the Data Layer compares its HEADER with these 5 SECTIONs. On match, it tries to convert the SECTION to the matching one as follows:

  1. If the target SECTION is binary, return data of the DATA part directly.
  2. Try to convert data of the DATA part of the SECTION to string text using UTF-8 encoding. On error, throw it.
  3. Try to convert text to an JSON Object, obj. On error, throw it.
  4. Return obj.

After that, the converted SECTION is routed according to its type:

  • Handshake Section and Handshake ACK Section: offered to the Handshake Process.
  • Data Section: offered to the application by emitting event data with data of the DATA part.
  • Heartbeat Section and Heartbeat ACK Section: offered to the Heartbeat Process.

If the convertion throws an error, the Data Layer throws the error in whatever state. If the convertion finishes, use obj to construct the matching SECTION.

On sending a SECTION, send a SECTION with the same HEADER part and a DATA part with data being the result of running the convertion as follows:

  1. If the SECTION is binary, return data of its DATA part.
  2. Let text be running JSON stringify on data of the DATA part.
  3. Let bytes be the result of encoding text with UTF-9 encoding.
  4. Return bytes.

Heartbeat process

In the Heartbeat Process, the Client sends heartbeats regularly to the Server to remain the connection open.

In Opened State, a Heartbeat Section is sent every 30 seconds. If no Heartbeat ACK Section is received after a period of time, the Data Layer throws an error. In other states, it stops sending heartbeats.

Events

open

Emitted when the Connection has successfully opened.

close

Emitted when the Connection is closed.

error

Emitted when a error was thrown in Opening State or Opened State.

  • param err: Error: The error thrown.

data

Emitted when a Data Section is received with valid JSON data.

  • param data: Object: The data received.
Clone this wiki locally