ID | IEP-86 |
Author | |
Sponsor | |
Created |
|
Status | IN PROGRESS |
One of the most important capabilities that Apache Ignite provides is the ability to colocate. There are two aspects of colocation:
Also Apache Ignite 1.x/2.x uses the term affinity key and a user must use affinity key in order to colocate the data. Because the main goal is data colocation, the user should not operate by affinity key term. Instead, the colocation term should be used directly.
As a typical example, let’s assume we have the following schema:
CREATE TABLE Accounts ( account_id INT, name VARCHAR, balance DOUBLE, PRIMARY KEY (account_id) ); CREATE TABLE Transactions ( tx_id INT, account_id INT, amount DOUBLE, PRIMARY KEY (tx_id) );
Most of the operations on such a schema will be isolated within a single account. Examples:
To achieve better performance, we want each of these operations to execute on a single node. Therefore, we need to group all transactions belonging to the same account within the same Raft group (partition, range, etc). Corresponding records that represent the account itself should also reside in the same group.
To achieve this, we want to perform affinity mapping for the Transactions
table using the account_id
field (as opposed to the tx_id
, as it would happen currently). We need to allow users to specify the field used for colocation. For example, like this:
CREATE TABLE Transactions ( tx_id INT, account_id INT, amount DOUBLE, PRIMARY KEY (tx_id, account_id) ) COLOCATE BY (account_id);
Note: The affinity key must always be a part of the primary key. Otherwise, an exception is thrown.
Note: Primary key columns, and therefore affinity key columns, MUST NOT permit NULL
values according to the SQL standard.
In case of using the composite primary key the tables also could be colocated. For example:
CREATE TABLE Orders ( order_id INT, department_id INT, create_date TIMESTAMP, PRIMARY KEY (order_id, department_id) ); CREATE TABLE OrderItems ( order_item_id INT, order_id INT, department_id INT, amount INT, PRIMARY KEY (order_item_id) );
For such data model the OrderItems
table could be colocated with the Orders
table in the following way:
CREATE TABLE OrderItems ( order_item_id INT, order_id INT, department_id INT, amount INT, PRIMARY KEY (order_item_id, order_id, department_id) ) COLOCATE BY (order_id, department_id);
Note: COLOCATE BY
clause of colocated table (OrderItems
table in the example above) must contain the same set of columns and in the same order as PRIMARY KEY
clause of the main table (Orders
table in the example above) in order to properly colocate the data.
Apache Ignite uses partitioning by a hash function. The hash function should have relatively even distribution. Because of it we can’t use primitive and very fast commutative hash functions (e.g. based on XOR
). So the hash function should respect the order of fields that are included into the affinity key.
Note: All APIs mentioned below are out of scope of data colocation feature design and will be designed later.
Note: this API will be defined later. Naming and ideas are not final.
Apache Ignite 3 will also provide the possibility to choose an affinity function for a group of some tables. At this point the term affinity [function] still should be used.
The only API that will be affected is the Compute API where affinityRun()
and affinityCall()
methods will be renamed to executeColocated()
(at the moment only one method is enough).
The affinity key mapper in Apache Ignite 1.x/2.x allows to implement some user defined logic for data colocation. Any user’s code is a potential problem because it leads to code deployment/redeployment issues, to error prone approach, etc. Instead of the affinity key mapper interface Apache Ignite 3 should provide the possibility to use a predefined set of SQL functions in the COLOCATE BY
clause (see for example PARTITION BY
clause in different databases).
Key | Summary | T | Created | Updated | Due | Assignee | Reporter | Priority | Priority | Priority | Priority | P | Status | Resolution |
---|
Key
Summary
T
Created
Updated
Due
Assignee
Reporter
Priority
Priority
Priority
Priority
P
Status
Resolution