Motivation

Implement thin Ignite client in any programming language / platform using a well-defined binary connectiona protocol.

"Thin" here means that we do not start an Ignite node in order to communicate with the cluster, we do not implement discovery/communication spi logic, as opposed to currently available Ignite Client Mode.

Description

TCP Socket

Every Ignite node listens on a TCP port. Thin client implementations connect to any node in the cluster through TCP socket and perform Ignite operations using a well-defined binary protocol.

Server-side connection parameters are defined in ClientConnectorConfiguration class. Default port is 10800. Connector is enabled by default, no configuration changes needed.

Data Format

Byte order is little-endian. Strings are UTF-8 with int prefix (no null terminator). All data types in this proposal are Java-like (byte = 8 bits, short = 2 bytes, int = 4 bytes, long = 8 bytes).

Binary Objects

User data, such as cache keys and values, is represented in Ignite Binary Object format. Binary Object can be a primitive (predefined type) or a complex object.

Primitives (predefined types)

Primitives are represented as a type code + value:

Available primitives are listed below:

Complex Objects

Complex objects consist of a 24-byte header, set of fields (binary objects), and a schema (field ids and positions).

Wrapped Binary Objects

One ore more binary objects can be wrapped in an array. This allows reading, storing, passing and writing objects efficiently without understanding their contents, performing simple byte copy.

All cache operations return complex objects inside a wrapper (but not primitives).

Message format

All messages, request and response, including handshake, start with int message length (excluding these first 4 bytes). E.g. empty message would be represented by 4 zero bytes.

Length is omitted in all message descriptions below for brevity.

Handshake

The first message upon connection establishment must be a handshake. Handshake ensures that client and server versions are compatible.

Client Operations

 Upon successful handshake client can start performing operations by sending a request with specific op code. Each operation has it's own request and response format, with a common header.

Operation codes and request ids are omitted everywhere below for brevity.

OP_CACHE_GET = 1000

Retrieves a value from cache by key.

int

Cache flags

GET and all other cache operations include flags byte.

OP_CACHE_GET_ALL = 1003

Retrieves multiple values from cache given multiple keys. 

int

OP_CACHE_PUT = 1001

Puts a value with a given key to cache (overwriting existing value if any).

int

Empy response.

OP_CACHE_PUT_ALL = 1004

Puts multiple key-value pairs to cache (overwriting existing associations if any).

int

Empy response.

OP_CACHE_CONTAINS_KEY = 1011

Returns a value indicating whether given key is present in cache.

int

OP_CACHE_CONTAINS_KEYS = 1012

Returns a value indicating whether all given keys are present in cache.

int

OP_CACHE_GET_AND_PUT = 1005

Puts a value with a given key to cache, returning previous value for that key.

int

OP_CACHE_GET_AND_REPLACE = 1006

Puts a value with a given key to cache, returning previous value for that key, if and only if there is a value currently mapped for that key.

int

OP_CACHE_GET_AND_REMOVE = 1007

Removes the cache entry with specified key, returning the value. 

int

OP_CACHE_PUT_IF_ABSENT = 1002

Puts a value with a given key to cache only if the key does not already exist. 

int

OP_CACHE_GET_AND_PUT_IF_ABSENT = 1008

Puts a value with a given key to cache only if the key does not already exist.  

int

OP_CACHE_REPLACE = 1009

Puts a value with a given key to cache only if the key already exists. 

int

OP_CACHE_REPLACE_IF_EQUALS = 1010

Puts a value with a given key to cache only if the key already exists and value equals provided value. 

int

OP_CACHE_CLEAR = 1013

Clears the cache without notifying listeners or cache writers.

int

Empy response.

OP_CACHE_CLEAR_KEY = 1014

Clears the cache key without notifying listeners or cache writers.

int

Empy response.

OP_CACHE_CLEAR_KEYS = 1015

Clears the cache keys without notifying listeners or cache writers. 

int

Empy response.

OP_CACHE_REMOVE_KEY = 1016

Removes an entry with a given key, notifying listeners and cache writers.

int

OP_CACHE_REMOVE_IF_EQUALS = 1017

Removes an entry with a given key if provided value is equal to actual value, notifying listeners and cache writers. 

int

OP_CACHE_GET_SIZE = 1020

Gets the cache size.  

int

OP_CACHE_REMOVE_KEYS = 1018

Removes entries with given keys, notifying listeners and cache writers. 

int

Empty response.

OP_CACHE_REMOVE_ALL = 1019

Removes all entries from cache, notifying listeners and cache writers. 

int

Empty response.

OP_CACHE_CREATE_WITH_NAME = 1051 

Creates a cache with a given name. Cache template can be applied if there is '*' in the cache name. Throws exception if a cache with specified name already exists. 

string

Empty response.

OP_CACHE_GET_OR_CREATE_WITH_NAME = 1052

Creates a cache with a given name. Cache template can be applied if there is '*' in the cache name. Does nothing if the cache exists.

string

Empty response.

OP_CACHE_DESTROY = 1056

Destroys cache with a given name.

int

Empty response.

OP_CACHE_GET_NAMES = 1050

Gets existing cache names.

Empty request.

string * count

OP_CACHE_GET_CONFIGURATION = 1055

Gets cache configuration

Cache configuration is sent by server in the following format:

OP_CACHE_CREATE_WITH_CONFIGURATION = 1053

Creates cache with provided configuration. Throws an exception if the name is already in use. 

Any number of configuration properties can be provided. Name is required.

Cache configuration data is specified in key-value form, where key is `short` property id and value is property-specific data. Table below describes all available properties.

Empty response.

OP_CACHE_GET_OR_CREATE_WITH_CONFIGURATION = 1054

Creates cache with provided configuration. Does nothing if the name is already in use. Same request format as above. Empty response.

OP_QUERY_SQL = 2002

Performs SQL query. 

Response includes first page of results.

OP_QUERY_SQL_CURSOR_GET_PAGE = 2003

Retrieves next SQL query cursor page by cursor id from OP_QUERY_SQL. 

OP_RESOURCE_CLOSE = 0

Closes a resource, such as query cursor.

Empty response.

OP_QUERY_SQL_FIELDS = 2004

Performs SQL fields query

OP_QUERY_SQL_FIELDS_CURSOR_GET_PAGE = 2005

Retrieves next query page.

OP_GET_BINARY_TYPE_NAME = 3000

Gets the platform-specific full binary type name by id. For example, .NET and Java can map to the same type Foo, but classes will be Apache.Ignite.Foo in .NET and org.apache.ignite.Foo in Java.

Names are registered with OP_REGISTER_BINARY_TYPE_NAME.

OP_GET_BINARY_TYPE = 3002

Gets the binary type information by id.

OP_REGISTER_BINARY_TYPE_NAME = 3001

Registers the platform-specific full binary type name by id. For example, .NET and Java can map to the same type Foo, but classes will be Apache.Ignite.Foo in .NET and org.apache.ignite.Foo in Java.

Empty response.

OP_PUT_BINARY_TYPE = 3003

Registers binary type information in cluster.

 Empty response.

OP_QUERY_SCAN = 2000

Performs scan query.

OP_QUERY_SCAN_CURSOR_GET_PAGE = 2001

Retrieves next SQL query cursor page by cursor id from OP_QUERY_SCAN. 

Message Exchange Example

As an example let's see how to connect to Ignite instance on a local machine and retrieve an int value for a given int key, in pseudocode.

var socket = openSocket("127.0.0.1:10800");
 
// Message length
socket.writeInt(8);
 
// Handshake operation
socket.writeByte(1);
 
// Protocol version 1.0.0
socket.writeShort(1);
socket.writeShort(0);
socket.writeShort(0);
 
// Client type: thin client
socket.writeByte(2);
 
// Receive result
var resLen = socket.readInt();
assert resLen == 1;  // Success message length is 1
var res = socket.readByte();
assert res == 1;  // Success code
// END HANDSHAKE
 
 
// Perform OP_CACHE_GET
// Message length
socket.writeInt(20)
 
// Op code = OP_CACHE_GET
socket.writeShort(1000);
 
// Request id (can be anything; does not matter for simple blocking socket IO)
var reqId = 1;
socket.writeLong(reqId);
 
// Cache id
var cacheName = "myCache";
var cacheId = javaHashCode(cacheName);
socket.writeInt(cacheId);
 
// Flags = none
socket.writeByte(0);
 
// Cache key 
socket.writeByte(3);  // Integer type code
socket.writeInt(1);  // Our cache value
 
// Read result
var len = socket.readInt();
assert len == 17;   // requestId (8) + status code (4) + int object (1 + 4)
 
var resReqId = socket.readLong();
assert reqId == resReqId;
 
var statusCode = socket.readInt();
assert statusCode == 0;  // Success
 
var resTypeCode = socket.readByte();
assert restypeCode == 3;  // Integer
 
var res = socket.readInt();
print(res);  // Resulting cache value

Complex Binary Object Example

An example of writing a custom object with a single integer field, in pseudocode:

// Header
socket.writeByte(103);  // Binary object type code
socket.writeByte(1);  // Binary object format version
socket.writeShort(1 + 2);  // Flags, user type + has schema
socket.writeInt(1512523596);  // Type id, Java-style hash code of "MyType"
socket.writeInt(0);  // Hash code, 0 for simplicity
socket.writeInt(24 + 5 + 8);  // Object length, header + int field (byte type + 4 bytes value) + schema (4 bytes id + 4 bytes offset)
socket.writeInt(0);  // Schema id, for simplicity
socket.writeInt(24 + 5);  // Schema offset, header size + one int field
 
// Fields
socket.writeByte(3);  // Field type, int
socket.writeInt(42);  // Field value
 
// Schema
socket.writeInt(1515208398);  // Field id, Java hash of "myfield"
socket.writeInt(24);  // Field position from object start, right after 24-byte header

Reading binary object and retrieving field value:

// Header
assert socket.readByte() == 103;
assert socket.readByte() == 1;
assert socket.readByte() == 1 + 2;
var typeId = socket.ReadInt();
var hash = socket.readInt();
var len = socket.ReadInt();
var schemaId = socket.ReadInt();
var schemaOffset = socket.ReadInt();
 
// Read field data into array, everything between header and schema
var fieldData = socket.readByteArray(schemaOffset - 24);
 
// Read schema
var fieldCount = (len - schemaOffset) / 8;
var fieldIds = new int[fieldCount];
var fieldOffsets = new int[fieldCount];
 
for (int i = 0; i < fieldCount; i++)
{
  fieldIds[i] = socket.readInt();
  fieldOffsets[i] = socket.readInt();
}
 
// Let's suppose that user wants a value of the field named "myfield"
var fieldName = "myfield";
var fieldId = getJavaHash(fieldName);  // 1515208398
var fieldIdx = fieldIds.indexOf(fieldId);  // Can be missing, handle accordingly
var fieldOffset = fieldOffsets[fieldsIdx];
 
// We got the field offset, now read binary object from fieldData
var reader = new ArrayReader(fieldData);
reader.seek(fieldOffset - 24);
 
var fieldType = reader.readByte();
if (fieldType = 3)  // int
{
  var fieldVal = reader.readInt();
}
else if (fieldVal = 103)
{
  // Nested binary object
  ...
}
else ... 

Discussion Links

http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-protocol-message-format-td20300.html

Reference Links

Similar protocols from other vendors:

• https://docs.mongodb.com/manual/reference/mongodb-wire-protocol/
• http://docs.hazelcast.org/docs/protocol/1.0-developer-preview/client-protocol.html