Protobufs

Overview

Protocol Buffers (protobufs) are a messaging protocol made by Google that allows for efficient encoding and decoding of complex data over a data stream (e.g. a network connection). The reason you need a protocol is that whenever you send data, both sides need to agree on the format of the data. We also want to send the minimal amount of information required to convey our message, (if you want to learn more about this field called information theory take EECS 126). Protobufs allow us to do both of these things easily.

It works by first defining a set of messages, similar to C structs, that can be sent. The definitions are done in the proto language which is explained in depth here https://developers.google.com/protocol-buffers/docs/proto3.

Then, in each language you need to create bindings from the .proto files to objects in that language. You'll notice that there's no option to choose C as a target language. Luckily, a group of people made a third-party extension of Google protobufs that can take .proto files and generate C source and header files that can be used for a C program to pack and unpack protobuf messages. A good place to understand how that works is the protobuf-c Github and their Wiki.

There are a few reasons why we chose Google Protocol Buffers for our communication with Dawn and Shepherd:

• Consistency with the old version of Runtime (old Runtime and Dawn used protobufs as well to communicate)
• Speed. It was decided the speed boost gained by using protobufs was worth the trouble of setting it up to work in C. Consider this: suppose we chose a protocol like JSON. To send a boolean, we might need to send the string: "{"switch0":false}". That's 19 characters (i.e. 19 bytes) sent over the network. Compare with protobufs, which would be literally 1 byte (0). Integers and complicated nested message types offer similar amounts of message size reductions. Combine it all together, and the network traffic reduction gained by using protobufs is substantial.
• Consistency between Runtime communication with Dawn and Shepherd. Shepherd uses JSON internally to send data around, and originally the plan was to communicate with Dawn using protobufs and communicate with Shepherd using JSON. But that's not very smart, because then Runtime would have to convert our internal data into two different message formats, which would be extremely ugly. So the decision was made to use protobufs for all of Runtime's network communications.

Tutorial & Usage

Below is a tutorial on using the protobuf-c library and the protobufs that the proto compiler generates. An incomplete tutorial can be found on the protobuf-c Github repo's Wiki, but these examples were what the original authors of Runtime used to learn how to use the library.

#include <stdio.h>
#include <stdlib.h>
#include "../pbc_gen/run_mode.pb-c.h"
#define MAX_MSG_SIZE 1024

static size_t read_buffer(unsigned max_length, uint8_t* out) {
    size_t cur_len = 0;
    size_t nread;
    while ((nread = fread(out + cur_len, 1, max_length - cur_len, stdin)) != 0) {
        cur_len += nread;
        if (cur_len == max_length) {
            fprintf(stderr, "max message length exceeded\n");
            exit(1);
        }
    }
    return cur_len;
}

int main() {
    RunMode* run_mode;

    // Read packed message from standard-input.
    uint8_t buf[MAX_MSG_SIZE];
    size_t msg_len = read_buffer(MAX_MSG_SIZE, buf);

    // Unpack the message using protobuf-c.
    run_mode = run_mode__unpack(NULL, msg_len, buf);
    if (run_mode == NULL) {
        fprintf(stderr, "error unpacking incoming message\n");
        exit(1);
    }

    // display the message's fields.
    printf("Received: mode = %u\n", run_mode->mode);  //comes in as unsigned int

    // Free the unpacked message
    run_mode__free_unpacked(run_mode, NULL);
    return 0;
}