Changes proposed in IEP-61 allows us to lower requirements for consistency guarantees of membership/discovery protocols. Current Discovery SPI implementations are not scalable enough or rely on external distributed systems (ZooKeeper), so relaxing consistency guarantees and improving scalability is a beneficial goal.
Other goals are:
Entry point class for all configuration of network module is NetworkConfiguration in org.apache.ignite.configuration.schemas.network package. It is an auto-generated class so project building is necessary to find it (configuration framework in greater details is described in IEP-55).
Right now we have to provide configuration for two main sub-components:
Main interface of network module is ClusterService. It provides access to two other aspects of network: discovery and P2P communication.
Discovery information is available via methods of TopologyService interface. Information is provided about local node and current set of nodes presented in the cluster, also it is possible to subscribe to events regarding topology changes like new nodes joining or existing nodes leaving.
P2P communication subsystem can be accessed through MessagingService interface. Here could be found various methods to send messages to other nodes providing different levels of guarantees about messages delivery.
Handlers for different types of messages are also registered in MessagingService.
This section provides a description of arbitrary objects serialization mechanism which allows network module to handle custom user objects. Unlike NetworkMessage descendants, user objects are not known to the system in advance, and serialization layout must be resolved at runtime.
User objects serialization protocol aims to implement the following properties and operates under the following assumptions:
A class descriptor represents a sequence of field names and field types that is used during class instances serialization and deserialization. Since the class descriptor is fully determined by the class itself, the descriptor is an immutable structure and can be identified by a unique descriptor ID.
Upon creation, each descriptor is assigned a unique identifier by the local node. Descriptors for the same class name may have different identifiers on different nodes.
Additionally, the descriptor contains a set of flags that specify which JDK serialization methods are used (readExternal/ writeExternal, readResolve/writeReplace, etc), and final flag for final classes.
Internally, Ignite will have a set of built-in descriptors that correspond to objects with fixed serialization format that includes, but not limited to:
Predefined descriptors allow for a more efficient serialization without compromising cross-version compatibility. Note that primitives and boxed primitives descriptors are effectively final.
Fields of the class are enumerated in a strict order from parent class to the child class, and sorted lexicographically within each class hierarchy level. If an Externalizable class is encountered (no matter at which level of hierarchy) it is serialized via writeExternal(); call.
During the class instance (de)serialization, the (de)serializer traverses the fields from the descriptor and writes:
If a field value is not a primitive its Object ID is also serialized to handle cycles of object references and support polymorphic fields.
Externalizable classes delegate the (de)serialization logic to readExternal/ writeExternal methods. The size of the serialized output should be made available to the general layout so that the Externalizable class can be skipped entirely in the case when the class on the deserializing side does not implement Externalizable interface.
In order to support arbitrary class structure changes, the object serialization must be performed according to the local class descriptor, but the object deserialization must be performed according to the remote class descriptor from the node that actually serialized the object. Therefore, the descriptors must be shared between Ignite nodes to support the protocol.
Descriptor availability is tracked on a per-session p2p level. If there is a notion of a session between nodes, the sending side can track which descriptors were already sent to the remote side before actually sending the serialized object. If there are unsent descriptors, they are sent to the remote side prior to sending the serialized object. The receiving side must use the descriptor from the particular session to deserialize the object. In this case, even if two different nodes send the class with the same name and different structure, the receiving side will be able to properly deserialize the object.
When reading a serialized instance using a remote class descriptor, the read values will not necessarily be present in the local class descriptor (this can happen, for example, if the local node removed/renamed a field from the class, or the remote node added/renamed a field). Since the layout of the read object precisely matches the remote class descriptor, the unexpected value can always be read and either skipped or passed to an optional handler.
If a field was added in the local class descriptor, but was not present in the remote class descriptor, it will be skipped during the deserialization. Such skipped fields can be additionally handled in the optional handler.
Netty - is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. It will be used for interaction with network(read/write, serialize/deserialize).
Scalecube - is a lightweight decentralized cluster membership, failure detection, and gossip protocol library. It will be used for gathering open network interfaces to cluster membership via gossip protocol.
Risks and Assumptions