One of the most important capabilities that Apache Ignite provides is the ability to colocate. There are two aspects of colocation:
Also Apache Ignite 1.x/2.x uses the term affinity key and a user must use affinity key in order to colocate the data. Because the main goal is data colocation, the user should not operate by affinity key term. Instead, the colocation term should be used directly.
As a typical example, let’s assume we have the following schema:
Most of the operations on such a schema will be isolated within a single account. Examples:
To achieve better performance, we want each of these operations to execute on a single node. Therefore, we need to group all transactions belonging to the same account within the same Raft group (partition, range, etc). Corresponding records that represent the account itself should also reside in the same group.
To achieve this, we want to perform affinity mapping for the
Transactions table using the
account_id field (as opposed to the
tx_id, as it would happen currently). We need to allow users to specify the field used for colocation. For example, like this:
Note: The affinity key must always be a part of the primary key. Otherwise, an exception is thrown.
Note: Primary key columns, and therefore affinity key columns, MUST NOT permit
NULL values according to the SQL standard.
In case of using the composite primary key the tables also could be colocated. For example:
For such data model the
OrderItems table could be colocated with the
Orders table in the following way:
COLOCATE BY clause of colocated table (
OrderItems table in the example above) must contain the same set of columns and in the same order as
PRIMARY KEY clause of the main table (
Orders table in the example above) in order to properly colocate the data.
Apache Ignite uses partitioning by a hash function. The hash function should have relatively even distribution. Because of it we can’t use primitive and very fast commutative hash functions (e.g. based on
XOR). So the hash function should respect the order of fields that are included into the affinity key.
Note: All APIs mentioned below are out of scope of data colocation feature design and will be designed later.
Note: this API will be defined later. Naming and ideas are not final.
Apache Ignite 3 will also provide the possibility to choose an affinity function for a group of some tables. At this point the term affinity [function] still should be used.
The only API that will be affected is the Compute API where
affinityCall() methods will be renamed to
executeColocated() (at the moment only one method is enough).
The affinity key mapper in Apache Ignite 1.x/2.x allows to implement some user defined logic for data colocation. Any user’s code is a potential problem because it leads to code deployment/redeployment issues, to error prone approach, etc. Instead of the affinity key mapper interface Apache Ignite 3 should provide the possibility to use a predefined set of SQL functions in the
COLOCATE BY clause (see for example
PARTITION BY clause in different databases).