Ignite lacks explicit support for single cluster multi DC (SCMD) deployment.
It's a significant feature gap when compared to other competitors.
Prevents Ignite from wider cloud adoption and restrict usage scenarios.
This article is a DRAFT of a features required to support efficient SCMD deployment.
It's known fact that cross DC network interaction has significantly worse round trip time then inter DC.
So the key idea is to add features and extension points that enable mostly inter DC communication in SCMD scenarious:
- Cluster info
- We must add some property to allow user to configure which DC node belongs to.
- This property must included in node info known cluster-wide.
- Discovery SPI
- TcpDiscovery must minify cross DC communication by forming discovery ring with only two cross DC edges.
ZooKeeperDiscovery has it's own guarantees related to ZK cluster.
- Affinity
- We must allow to the user to setup two different Affinities:
- multi DC - when cache partitions must be spread to as many DC as possible to overcome DC outages.
- Do we need to support "single DC" configuation? Do we have real-world use-cases for it?
- Queries
- FULL_SYNC and readFromBackupt=true is a primary setup here.
- Key-value API must go to the same DC nodes whenever possible.
- From the first glance, looks like it's possible to read always from
- SQL - calcite engine must maps queries fragments to the same DC nodes whenever possible.
- Client nodes
- Must connect to the same DC server nodes.
- Thin clients
- Must support same DC read key-value queries.
- Must select same DC server node for a node-binded communcations: transactions, tasks, etc
- Rebalance
- Partitions must be loaded from same DC nodes whenever possible.
- Snapshot restoration
- When restore on different topology must load partitions from same DC nodes whenever possible.
- Configuration
- network, query, transaction and other timeouts must be set with respect to cross dc latency.
Topology validator
Reed solomon codes to reduce replication factor support - Testing
- Standard setup - 3 dc + 5 backups? must be carefully tested.