Status
Current state: Accepted.
Discussion thread: here
Vote thread: here
JIRA: KAFKA-3487
Released: 0.11.0.0
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka Connect is a highly extensible framework, designed to run software modules that integrate a wide variety of data sources and destinations with Kafka and, by definition, are developed independently of the framework. Examples of such software modules are connectors, transformations and data converters, which hereafter will be called plugins to honor Kafka Connect terminology. Because such independent development, dependency conflicts might arise naturally, between the framework and the plugins, as well as among the plugin themselves. This happens when, for instance, one connector is using the latest version of a package that added a new feature, while another connector depends on a previous version of the same package where this feature is missing.
The effect is not very severe because software packages appearing often as common dependencies are usually doing a good job in preserving software compatibility between versions. Thus, such issues can usually get resolved with appropriate ordering of packages in the classpath. Furthermore, enough flexibility has been demonstrated so far between framework and connector developers in syncing up with updates of commonly used dependencies.
Nevertheless, as Kafka Connect adoption grows, runtime exceptions due to version mismatches might arise more often, and could become a burden in the development and deployment of connectors with Kafka Connect.
In order to resolve the issue of conflicting dependencies, class loading isolation is suggested here as a remedy.
In this case, the framework will do the heavy lifting in terms of development, offering a transparent resolution of dependency conflicts and keeping the changes in Kafka Connect's public interfaces minimal.
Public Interfaces
The only publicly visible change that is proposed in order to implement class loading isolation in Kafka Connect is the addition of a new Connect worker config property:
This is a framework level configuration property that will affect all the connectors running within a Connect worker.
The new configuration property plugin.path will accept a list of locations, represented as strings and separated by commas. The strings representing locations should be able to be transformed into URLs in order to offer maximum flexibility in terms of plugin discovery. Nevertheless, in the first implementation of class loading isolation such locations will be paths to the local filesystem, that, in turn, can be easily transformed into URLs.
Examples of valid values for plugin.path:
- plugin.path=/usr/local/share/java
- plugin.path=/usr/share/java, /usr/local/share/java, /opt/connectors
When a filesystem path is used as location for imported plugins (connectors, transformations and converters), the convention is that each plugin is storing all its required dependencies in a single directory under each location path listed in plugin.path. For instance, if the location /usr/local/share/java is given in plugin.path then plugins such as my-kafka-source-connector, my-kafka-sink-connector, my-connect-smt and my-converter should store all of their dependencies (usually in the form of jar files, but possibly as raw class files with the appropriate java package structure) under directories immediately below each plugin.path entry as follows:
/usr/local/share/java
/my-kafka-source-connector/(connector jars and dependency jars)
/my-kafka-sink-connector/(connector jars and dependency jars)
/my-connect-smt/(smt jars and dependency jars)
/my-converter/(converter jars and dependency jars)
A more nested storing of plugin and dependency jars will not make them discoverable.
Again with plugin.path set to /usr/local/share/java an incorrect example would be:
/usr/local/share/java
/more-plugins/another-kafka-source-connector/(connector jars and dependency jars)
In this case, in order to make this plugin discoverable, /usr/local/share/java/more-plugins should be added to plugin.path instead.
Additionally, specifically regarding Connect plugins, each connector, transformation or converter jar should be listed only once across the plugin path. If multiple copies of the same version of plugins exist under plugin.path the selection of which plugin will be loaded will be deterministic but implementation specific.
While the introduction of the new configuration property plugin.path is the only change proposed to the public interfaces of Kafka Connect, next are also described the main implementation steps that will be carried out within the framework to support class loading isolation.
Proposed Changes
Add plugin.path configuration property for workers running both in standalone and distributed mode. By default plugin.path is empty. In this case isolation is not active and loading of connectors depends on the CLASSPATH. In general, when a user wants to run a connector is isolation, its packages along with its dependencies should be stored under a directory that is listed in plugin.path. Otherwise if non-isolated execution is desired, the connector jars should be listed in the CLASSPATH.
Given that plugin.path is set appropriately, the Connect framework will be able to instantiate a custom plugin classloader for each plugin under the list of locations supplied in the plugin.path. The main characteristics of such a plugin classloader are that:
- it delegates loading of Connect framework classes as well as java library classes to the appropriate classloader. Optionally, warnings could be issued when a plugin path contains Connect framework classes.
- it applies a child-first policy for the rest of the classes, aiming to load plugin specific dependencies directly.
Furthermore, the framework will control which classloading policy is in effect every time, by setting appropriately the thread context classloader (TCCL). This is not expected to be an issue, even in environments that use OSGi or other module deployment frameworks for two reasons:
- the Connect framework controls the threads that run plugin code (e.g. connector tasks).
- plugin classes and dependencies are required to be supplied explicitly through plugin.path.
Compatibility, Deprecation, and Migration Plan
- Existing users will not be impacted since an non-existent/empty plugin.path means that isolation is not in effect.
- Users enabling class loading isolation might experience higher demands in memory usage due to additional loading of otherwise common classes. However this increase is not expected to be prohibitive in most cases.
Test Plan
- Targeted Unit tests will be developed to test the components that will implement class isolation in the framework. Additionally, all other tests will be set to run with class loading isolation turned on.
- System tests will be written to test class loading isolation when explicitly conflicting dependencies are introduced by connectors.
- Microbenchmarks will be designed to make sure the effect of classloader context switching is negligible.
Future Work
In a subsequent version of class loading isolation, the following enhancements will be targeted:
- Support versioned plugins. Users will have the ability to load and run different versions of the same plugin. Although connectors are versioned already, this capability needs to be enabled for transformations and converters as well.
- Extend plugin.path to support plugin locations beyond the local filesystem. Such locations might include web addresses, packages repositories (e.g. maven repositories) and other locations.
Rejected Alternatives
Use OSGi. OSGi. Along with its much broader scope, an OSGi implementation would bring significant implementation complexity, both in the framework and the connector development.
- Design for project Jigsaw and wait. With this KIP we describe an implementation path whose execution is expected to be focused and efficient. Major upgrades, such as an upgrade to the next Java version are not expected to move faster than the proposed implementation.