Network protocol design is hard to get exactly right, and networking/serialization code has high standards of correctness – all the other code in a project depends on the right data getting sent across the wire. If you're trying to communicate between programs written in multiple languages, that code has to be written (and tested, and maintained) separately in each language.

RPC frameworks offer a nice solution to this problem: write the interface you want and your message definition in some Interface Definition Language (IDL), and then compile it to working (and hopefully idiomatic) code in a multitude of languages.

However, there are drawbacks:

  • We don't control the implementation, which means it is hard to optimize for performance, and harder to add new features later.
  • many of the RPC frameworks don't support push notifications from the server to client (implemented by some as "streaming").
  • Generally, IDLs require static type definition. This means that we would have to define remote procedure calls as only taking one type. The only really workable approach would be to take a byte array serialized with some other data serializer and pass that through the RPC. This reduces the usefulness of an RPC framework somewhat, as it can't handle everything. However, having function stubs and having the transport taken care of is still quite useful, especially as the code works in multiple languages.

 

"Server-Client push" in the table below refers to the ability of the server to push messages back over the same communication channel the client uses to connect to it. BERT, for example, can take an address to deliver a message to when some remote call is done, but it has to be able to connect to that server. Many Geode users use firewalls, which would cause issues connecting to the client. Also, it's more adapted to the actor model than to a client-server model, as we use.

RPC FrameworkSupported LanguagesSerializationTransport LayerServer-Client push?Dynamic type definition?
Avroat least Java, Python, Ruby, C#, C++Binary or JSONtransport-agnosticNoYes
BERTscroll to the bottom on the website to see. NO native Java support (though there is Scala).Erlangtransport-agnostic or BERP (custom)NoNo
gRPC10 (incl. C++,Java,C#,Go)ProtobufHttp2, Java uses NettyYesNo

Apache Thrift

quite a few (incl. C++,Java,C#, (unofficial) Go)CustomPluggable: TTransport interface.NoNo

Raw Socket with

Serialization Framework

See supported languages of the serialization frameworkProtobuf, Avro, Cap n ProtoSocketChannelNoNo

Testing Setup

All perf tests were run on a  4x 2.4Ghz (4-Core), 48Gb RAM.

Two sets of performance tests were run:

  • 1x server with 1 client doing 10mil blocking put messages
  • 1x server with 2 clients doing 10mil blocking put messages each

Each test was run with the same message structure and key/value size.

Key Size: 64bytes

Value Size: 512bytes

The put message definition for each framework are linked in the table:

FrameworkIDL Message definition files
Avro (using raw socket)client.avdl
Cap n Proto (using raw socket)protocol.capnp
Protobuf (using raw socket)clientProtocol.proto region_API.proto
GRPC + ProtobufgrpcService.proto clientProtocol.proto region_API.proto 

 

The results can be found in Results.zip. This archive file contains all the results for both testing scenarios, including GFS files for more detailed stats.

Results

1 Server, 1 Client (1 Thread)

 ProtobufAvroCap n ProtoGRPC StreamingGRPC Blocking
Median (msg/s)37593.9849621012.8178223245.0023247370.914266931.447979
Average (msg/s)37040.2269520541.2210923182.097745986.40686868.469895

 

 

1 Server, 2 Client (1 Thread)

 ProtobufAvroCap n ProtoGRPC StreamingGRPC Blocking
Median (msg/s)65307.391540355.2581843851.9591788949.444711974.32611
Average (msg/s)65471.258940375.6038643770.1022583513.3669411933.20999

 

 

 

 

  • No labels