Motivation

Ignite codebase and release packages mix both core capabilities with 3rd party integrations. It leads to the following:

• Cumbersome and continuously growing codebase with many 3rd-party dependencies. 
• Some of the integrations are questionable and should no longer be supported by the community at all.
• Integrations evolution is bound to Ignite release cycles even though no changes are needed in the core.
• Ignite community has to support everything (test, release, fix, continue development) which requires to have particular integration experts on a permanent basis - doesn't work.

The goal of the IEP is to solve these challenges by doing the following:

• Split existing Ignite codebase into the core and modules (key integrations to be supported by the Ignite community).
• Select key modules which the community will continue to test and support. Move other integrations out of community support/control.
• Ensure that modules can be released and evolve separately from the Ignite core.
• Modules will be in their own repositories that will let to support different versions of a product Ignite is integrated with via branching (for instance there is a demand to support different Spark versions, but with the present monolithic architecture it is challenging). 

Description

Below you can find a definition of Ignite core, list of modules to be supported by the community and integrations that will go out of the community control.

Ignite Core

Ignite core is a set of features and components that define the project's key capabilities and benefits such as a distributed memory-centric storage, RDBMS acceleration, transactions, and more. Most of such components are developed from scratch by the community and have minimal dependencies on 3rd parties:

• Memory-Centric Storage, Native Persistence, RDBMS Integration (CacheStore for RDBMS)
• Key-Value APIs (NO JCache support)
• SQL
• Compute Grid
• Machine Learning APIs
• Advanced queries -scan, continuous, full-text search
• Transactions
• Data Structures and Atomics
• Ignite Messaging
• Core Streaming APIs such as IgniteDataStreamer
• Logging
• Metrics and Tracing Framework
• Command line tools and scripts such as Visor and control.sh
• Standard (aka. thick clients) - Java, .Net, C++.
• Spring Core - needed for configuration needs.

Ignite Modules

Ignite modules are integrations that will be developed, supported, and released by Ignite community. Such integrations are important ones and have either significant or growing demand. Presently, the list is as follows:

• Spark Integration
• SpringData and SpringBoot
• Kafka Integration
• Service Grid
• TensorFlow Integration
• Cassandra Integration

Requirements and Installation

Each module has to satisfy the following criteria:

• Each module should be located in a separate GitHub repository.
• A module can be released separately from the core.
• A module has to be tested with existing testing tools like TeamCity.
• Each module is tested for every Ignite core release. A new version of a module is released together with the core release if changes are required.
• Modules versioning doesn't need to be aligned with the core. A compatibility matrix between the core and a will be maintained.

A module needs to be released and packaged in the form of:

• Maven/Nuget/Npmjs or another artifact depending on a programming language.
• Docker image (for selected plugins, to be decided on a case by case).
• Separate binary downloadable.

Even though the modules are released and packaged separately, there has to be an easy way to move modules' binaries to the Ignite core folder:

• Users downloads ignite-module-x plugin (with all of the JARs needed)
• A script like that is executed to add it to the Ignite core folder - ./ignite_plugin.sh install /Downloads/ignite-module-x
• ignite-module-x libs/JARs are installed into IGNITE_HOME/libs

Thin Clients

Thin clients are a special type of Ignite modules:

• Thin Clients for the following programming languages - Java, .Net, C++, Node.JS, Python, PHP.
• JDBC and ODBC drivers

Thin clients should be stored in separate repositories and packaged in different forms - binaries, Maven/Nuget/Npmjs (depending on a language type).

Integrations With No Community Support

Below is a list of existing integrations that won't be turned into Ignite modules but rather would be moved to separate Github repositories and won't be maintained by Ignite community for every core release. If later the community sees demand for an unsupported integration, it can be taken back and be officially supported (testing, dev, releases, compatibility with the core) as an Ignite module.

The list of integrations that won't longer be supported by the community:

• Twitter, ZeroMQ, RocketMQ, Storm, Flume, Flink, MQTT, Camel, Hibernate, JMS, AOP-Based Grid
• OSGi, YARN, Mesos

Following modules have to be discontinued and removed at all:

• Redis and Memcached protocols support
•  ignite-clients module (we already have Thin Clients, duplicate features with fewer capabilities)
• Hadoop Accelerator and IGFS

Risks and Assumptions

• Challenges with a modification of existing build and testing procedures
• Release policies have to be updated to ensure that modules & core versions compatibility matrix is updated regularly. 

Discussion Links

http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html

Reference Links

This initiative is also related to the discussion of Apache Ignite APIs update/removal - Apache Ignite 3.0 Wishlist

Tickets

// Links or report with relevant JIRA tickets.