Apache Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures. While it currently has high-speed client interfaces for Java, C++ and .NET there is both a need to create lighter-weight clients and a demand to access Geode from other programming languages. Unfortunately, the existing client-server protocol is undocumented. It evolved over time and is overly complex to meet either of these needs.
This proposal details the requirements, API and structure for a new client/server protocol. It does not specify the exact serialization mechanism, but the intent is for the protocol that is described to be complete in terms of the interface and message ordering. The intent is to make it pluggable so that we can experiment with different serialization formats based on varying performance and ease-of-use needs. In particular, we expect to use a widely-available IDL to serialize the protocol at first and make it accessible from many languages, and possibly implement a custom serialization later for clients needing very high performance. Choosing the IDL is an open goal.
The intent is to allow client functionality to be implemented in phases, moving from a "basic client" to a more advanced "smart client". It endeavors to provide a protocol that is also more amenable to more modern APIs such as those using asynchronous or reactive patterns.
Serialization of application keys, values, callback arguments, function parameters and so forth is a separate matter and are not necessarily tied to the serialization protocol used for client/server messaging. The initial protocol will support primitive types such as scalars, strings, and byte arrays. It will also support JSON documents as values and convert between these and Geode PDX-serialized objects in the servers.
The high-level goals for the protocol are defined here.
In the evaluation or definition of any protocol, it expected that the evaluated protocol/framework meets the following requirements:
Versioning: The protocol has to provide version information, in order to distinguish different protocol versions from one another.
Correlation Id: This number is a unique identifier that allows the system to correlate requests and responses.
Object Type: The serialization type of the data objects stored inside the messages
Response Type: It indicates whether a response is partial or complete.
ErrorCodes: It indicates the problem with API invocation.
Chunk Response: The ability to send a large response in multiple smaller, more manageable chunks.
Continuous Response: Client can register(Observer pattern) for events and then server notify the client if those events occur.
Request: The request message to be sent
Response: The response message received in relation to a request message
Request Format: Format of request API and its parameters, which client wants to invoke.
Response Format: Format for API return value, which client invoked.
Message: The generic construct that represents a message that is to be sent which contains a Message Header and Request/Response.
Serialized Byte Order: Big Endian
RPC and Message serialization Frameworks
During the investigation into frameworks to help "lower the barrier of entry," it became evident that there are two types of external frameworks:
- Message Serialization Frameworks - These frameworks allow for the definition of a message in a generic IDL, the generation of language specific classes from the IDL, and the encoding/decoding of those message to be sent over a transport
- RPC Frameworks - There frameworks provide greater coverage in the node-to-node communication: the transport layer (HTTP, TCP, UDP), the message definition in IDL with corresponding serialization mechanism and the definition of methods in the IDL, as well as the generation of corresponding service stubs.
- Message serialization frameworks define the encoding/decoding of defined messages but not the transport or connectivity.
- RPC frameworks concern themselves with connectivity and transport, remote method invocations and the encoding/decoding of defined messages.
Because this protocol needs to be tunable for very high performance, for some lack of functionality and because RPC frameworks hide their network and threading internals, it was decided that option 2 was not viable. See a comparison at RPC framework evaluation.
From an higher-level architectural perspective we can identify 2 layers:
- A Transport Layer (TCP, UDP, HTTP, etc..)
- Message encoding/decoding Layer
This proposal will define the message structure and protocol to be agnostic of transport used.
Message Structure Definition
All details relating to the Message structure definition can be found on the page Message Structure and Definition.
Proposed Implementation Phases
Introducing a new protocol into GEODE has the potential to be highly disruptive. In order to minimize the disruption and maximize the feedback cycles, we suggest implementing the changes in a phased approach. To view the milestones for each phase please see the page Phases and Milestones.
To better visualize the protocol messages a few sample messages have been provided on the page Protocol Message Examples