...

ID

IEP-76

Author

Pavel Tupitsyn

Sponsor

Pavel Tupitsyn

Created

23 Jun 2021

Status


colour	Green
title	ACtIVE

Table of Contents

Motivation

Thin client protocol will be the primary way to interact with Ignite 3.0 from code.

Description

Adapt Ignite 2.x protocol for Ignite 3.0. The main differences are:

MsgPack as underlying binary format (IEP-75: Thin Client MsgPack Serialization)
A different set of data types (IEP-54: Schema-first Approach)
More flexible and extensible handshake with magic header string

TCP Socket

Every Ignite node listens on a TCP port. Thin client implementations connect to any node in the cluster (possibly multiple nodes) through a TCP socket and perform Ignite operations using a well-defined binary protocol.
Server-side connection parameters are defined in ClientConnectorConfiguration class.
- Default port is 10800.
- Connector is enabled by default, no configuration changes are needed.
Netty is used on the server for network IO.

Data Format

MsgPack is used for data serialization (uses big-endian byte order).

Data Types Mapping

Ignite data types defined in IEP-54 map to MsgPack data types the following way:

...

"Extension X" is a MsgPack extension type (up to 128 extension types are allowed).

Integer data types, varint encoding

MsgPack provides multiple data types for integer values. When encoding a value, the smallest data type for that value is picked automatically.

"int" is used below to denote int format family. Values such as payload size, request ID, error codes, are encoded this way - using variable number of bytes based on the value.

Message format

All messages, requests and responses, except handshake, start with a 4-byte message length (excluding the length itself).

...

4 bytes	Length of Payload
...	Payload

Tuple serialization

This IEP covers Table API - Ignite#tables(), which is the only public API available at the moment.
Table API operates on tuples (Tuple and TupleBuilder interfaces).
Tuple is a set of key-value pairs - column name and value.
Values can be of basic types described above and in IEP-54, according to the table schema.
Table schema can evolve over time, columns get added and removed. Ignite tracks the evolution with incrementing integer schema version.
All schema versions are stored by Ignite. It is possible to retrieve the set of columns for any version.

...

Code Block

language	java
title	Tuple serialization (pseudocode)

for (Column col : tuple) {
	switch (col.type().spec()) {
    	case BYTE:
        	packer.packByte(tuple.byteValue(col.name()));
	        break;
	    case SHORT:
...

Key tuples

Key tuples are tuples with key columns only. Key columns always come first in the schema. So if there are 2 key columns, first two values of the tuple is the key.

Null vs NoValue

Ignite Table API handles "column set to null" (1) and "column not set" (2) differently.

...

NoValue custom protocol type reflects this distinction.

Handshake

Request
4 bytes	Magic number 49 47 4E 49, or "IGNI". Helps to identify misconfigured SSL or other apps probing the port.
int	Payload length
int	Version major
int	Version minor
int	Version patch
int	Client code (1 for JDBC, 2 for general-purpose client)
bin	Features (bitmask)
map (string → any)	Extensions (auth, attributes, etc). Server can skip unknown extension data (when client is newer).

...

Response
4 bytes	Magic number 49 47 4E 49, or "IGNI". Helps to identify misconfigured SSL or different server on the port.
int	Payload length
int	Server version major
int	Server version minor
int	Server version patch
int	Error code (0 for success)
string	Error message (when error code is not 0)
int	When error code is 0: Server idle timeout
string	When error code is 0: Server node id
string	When error code is 0: Server node name (consistent id)
bin	When error code is 0: Server features (bitmask)
map (string → any)	When error code is 0: Extensions (auth, attributes, etc). Client can skip unknown extension data (when server is newer).

Client Operations

Upon successful handshake, client can start performing operations by sending a request with specific op code. Each operation has its own request and response format, with a common header.

...

Request or response without operation-specific data is called basic request or basic response below.

Notifications

Clients should expect notification messages at any moment after the successful handshake.

...

Operation codes and request ids are omitted everywhere below for brevity.

TABLES_GET = 3

Basic request.

Response
map (UUID -> string)	map of table ids and names

Note: tables can only be created/deleted with SQL, there are no TABLE_CREATE or TABLE_DROP operations.

TABLE_GET = 4

Request
string	table name

Response
UUID or nil	table ID or null when a table with the given name does not exist

SCHEMAS_GET = 5

Request
UUID	table ID
arr or nil	schema IDs, or null to get latest

...

Response
map (int → array (array))	Map from schema ID to columns. Column is represented by an array of values for: name, type, isKey, isNullable. The array can contain extra data in future for additional properties.

TUPLE_UPSERT = 10

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

If columns don't match, request the latest schema and try again.
If the latest schema still does not match, and live schema is enabled, use TUPLE_UPSERT_SCHEMALESS

TUPLE_GET = 12

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for key columns (in schema order)

...

Clients should retrieve schemas with SCHEMAS_GET and cache them per table.

TUPLE_UPSERT_ALL = 13

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
int	row count
values	rows with values for all columns in given schema (nil when value is missing for a column)

Basic response.

TUPLE_GET_ALL = 15

Request
UUID	table ID
int or nil	transaction ID
int	schema ID, or nil when result set is empty
int	row count
values	array of rows with values for key columns (in schema order)

...

Response
int	schema ID (for all tuples in response)
int	row count
values	rows with values in schema order

TUPLE_GET_AND_UPSERT = 16

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

Response
int or nil	schema id for the current tuple, or nil when there is no matching record
values	values for value columns in schema order

TUPLE_INSERT = 18

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

Response
bool	Insert result

TUPLE_INSERT_ALL = 20

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
int	row count
values	rows with values for all columns in given schema (nil when value is missing for a column)

...

Response
int	schema id, or nil when no rows were skipped
int	skipped row count
values	skipped rows (values in schema order)

TUPLE_REPLACE = 22

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

Response
bool	Replace result

TUPLE_REPLACE_EXACT = 24

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	oldRec: values for all columns in given schema (nil when value is missing for a column)
values	newRec: values for all columns in given schema (nil when value is missing for a column)

...

Response
bool	Replace result

TUPLE_GET_AND_REPLACE = 26

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

Response
int or nil	schema id for the current tuple, or nil when there is no matching record
values	values for value columns in schema order

TUPLE_DELETE = 28

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for key columns in given schema

...

Response
bool	Delete result

TUPLE_DELETE_ALL = 29

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
int	row count
values	rows with values for key columns in a given schema

...

Response
int	schema id, or nil when no rows were skipped
int	skipped row count
values	skipped rows (values for key columns in schema order)

TUPLE_DELETE_EXACT = 30

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

Response
bool	Delete result

TUPLE_DELETE_ALL_EXACT = 31

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
int	row count
values	rows with values for all columns in given schema (nil when value is missing for a column)

...

Response
int	schema id, or nil when no rows were skipped
int	skipped row count
values	skipped rows (values for key columns in schema order)

TUPLE_GET_AND_DELETE = 32

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for all columns in given schema (nil when value is missing for a column)

...

Response
int or nil	schema id for the current tuple, or nil when there is no matching record
values	values for value columns in schema order, when schema id is not null

TUPLE_CONTAINS_KEY = 33

Request
UUID	table ID
int or nil	transaction ID
int	schema ID
values	values for key columns

...

Response
bool	whether a tuple with the given key exists

TX_BEGIN = 43

Request
bool	Read-only tx flag
long	Observable timestamp (value from server - see standard response header)

...

Response
int	Transaction ID

TX_COMMIT = 44

Request
int	Transaction ID

Basic response

TX_ROLLBACK = 45

Request
int	Transaction ID