Usage of gRPC and Protocol Buffers

[huakim]

Should we use gRPC and Protocol Buffers instead or alongside JSON and HTTP for server-client communications?

The network incurs money costs for both uploaders and server owners, and inefficient use of network resources also leads to time losses.

Combining WebSockets and Protocol Buffers can be a great choice, and here are some reasons to consider there technologies, as well as some potential drawbacks:

Reasons to Combine WebSockets and Protocol Buffers

Efficiency:

• Protobuf is more compact than JSON, which reduces the amount of data sent over the network. This is especially beneficial for applications that send a lot of data over HTTPS.

Performance:

• gRPC provide low-latency communication, and when paired with the efficient serialization of Protobuf, you can achieve fast data transmission.

Reduced Bandwidth:

• The combination of two techniques can help minimize network data usage and time costs.

I know that COPR is slower compared to the Open Build System, and I think we can enhance COPR by using resources more efficiently. I would like to contribute to the project by implementing Protocol Buffers and gRPC

[FrostyX]

Personally, I have never heard of gRPC and Protocol Buffers, and I only know about some web frameworks (Phoenix, Liverwire, etc) that use websockets to complement HTTP. I understood the reason for them doing so, as a way to create frontend-heavy websites without actually writing JavaScript. But maybe I misunderstood the point.

I know that COPR is slower compared to the Open Build System, and I think we can enhance COPR by using resources more efficiently.

100% agree Copr is kinda slow. Definitely slower than I would like it to be. But the problem is IMHO on the database layer or application layer. My bet wouldn't be on HTTP. Do you have some kind of benchmark that can be done to see the potential benefit of what you are proposing?

[huakim]

Protocol Buffers, or protobuf, is a data serialization format that allows define structured data in a concise way. It uses a .proto file to specify the data schema, which can then be compiled to generate code in various programming languages. This format is primarily binary data-based, meaning it encodes data in a compact binary format rather than human-readable text, making it efficient for both storage and transmission.

gRPC is a framework designed for building efficient remote procedure call (RPC) APIs, based on Protocol Buffers. It facilitates communication between services by defining service interfaces and message types using Protocol Buffers. gRPC is indeed based on HTTP/2, which allows it to leverage features like multiplexed streams and improved performance for network communications.

[huakim]

// This example demonstrates how to use a Protocol Buffers (protobuf) file to create a 
// gRPC-based server, showcasing the simplicity of the implementation process.
syntax = "proto3";

package org.fedora.copr;

option go_package = "fedora/copr";

message Creds {
    string login = 1;
    string username = 2;
    string token = 3;
}

message NewPackageRequest {
    Creds creds = 1;
    string project = 2;
    string name = 3;
    bool webhook_rebuild = 4;
    uint32 max_builds = 5;
    uint32 timeout = 6;
    BuildType type = 7;
}

message BuildType{
    oneof type {
        ScmType scm = 1;
        DistGitType distgit = 2;
        CustomType custom = 3;
    }
    message ScmType{
        string clone_url = 1;
        string commit = 2;
        string subdir = 3;
        string spec = 4;
        RepoType scm_type = 5;
        BuildMethod method = 6;
        enum RepoType {
            GIT = 1;
            SVN = 2;
        }
        enum BuildMethod {
            RPKG = 1;
            TITO = 2;
            TITO_TEST = 3;
            MAKE_SRPM = 4;
        }
    }
    message DistGitType{
        string commit = 1;
        string namespace = 2;
        string distgit = 3;
    }
    message CustomType{
        string chroot = 1;
        repeated string build_requires = 2;
        string resultdir = 3;
        repeated string repos = 4;
        string script = 5;
    }
}

message ActionResult{
    string text = 1;
    bool success = 2;
    optional ErrorCode = 3;
    enum ErrorCode{
        MALFORMED_REQUEST = 1;
        PACKAGE_EXISTS = 2;
        AUTHORIZATION_FAILED = 3;
        PROJECT_NOT_EXISTS = 4;
        PACKAGE_NOT_EXISTS = 5;
        PROJECT_EXISTS = 6;
        BUILD_FAILED = 7;
    }
}

service copr {
  rpc add_package(NewPackage) returns (ActionResult);
}

// This example underscores how straightforward it is to build a gRPC server with 
// protobuf, making it an ideal choice for efficient and scalable service development.
// We have to write "protoc grpc.proto --python_out=." to compile that, and implement 
// server logic on python, or, "protoc grpc.proto --go_out=." to compile that, and 
// implement server logic on  golang

[FrostyX]

The linked benchmark is an image from https://blog.shiftasia.com/grpc-vs-rest-speed-comparation/ ... I meant a benchmarking Copr itself.

[huakim]

No, i haven't yes

[FrostyX]

It is easy to run your own Copr instance locally https://frostyx.cz/posts/copr-docker-compose-without-supervisord
Feel free to experiment :-)