Page History

...

Our initial assumption is that all plugins are installed manually across the cluster. Installation here means copying the necessary files (e.g jars, modules, libraries) in the classpath.

A producer can select a compression plugin through configuration the same way it chooses a built-in codec. When the codec is not a built-in, the producer will consume the plugin list from the topic and retrieve the plugin information. 

...

The plugin information schema currently includes the following fields:

1. pluginID: an automatically generated integer value automated generated that represent the plugin identification number and shared across all the language clientclients. It is used to communicate the selected plugin to the broker rather than using the alias.
2. pluginAlias: a string value holding the name of the plugin and shared across all the language client. It is the value for the plugin used in compression.type configurations.
3. pluginClassName: a string value holding the class name to be instantiated when pluggable interface is used. The name here is java centric, but the idea is that for any other language, this is where the code for using the plugin will live
4. pluginVersion: a string value holding the version number of the plugin

...

Constraints for registering plugins and ensure compatibility :

1. All plugins entries for different language-client must be installed upfront ensuring they have the same pluginID. Registering them at the same time allows admin to ensure all clients are compatible.
2. Ensure no other java plugin with same alias does not already exist
3. When adding non java plugin, make sure the java plugin does exist so pluginID can be retrieve from the java versionNon java plugins may not be registered without the java equivalent because the broker needs to be able to decompress the data 
4. All plugin with same alias should must be compatible.
5. Update to an existing plugin that is not compatible must be If a plugin has a new alias then it is a new plugin

Message Format Update

...

The assumption at the moment is that the plugin is manually installed, . Automatic plugin installation is a stretch goal at this time.

Selecting plugins from Producer

...

1. A Kafka broker decodes the batch headers and the codecID is 5 which signals a plugin was used
2. The broker then decodes the pluginID
3. The broker consumes the plugin list from the plugin-java-topic and matches the pluginID to the plugin list.  
4. If the broker select a plugin which is not in the plugin list, then it will receive an "Unknown compression name" illegal argument exception.
5. The broker instantiates a plugin compressionType object then validates and analyzes the offsets.
6. On recompression, the broker will consume the plugin list and match the pluginAlias to the plugin list

Selecting plugins from Consumer

...

1. The current proposal does not have any impact on existing users. 
2. We are not changing the behavior of current compression. We are simply adding a new compression interface. If message format changes in Kafka then this implementation will need to be updated
3. Regression testing shows that pluggable interface did not affect default performance using Snappy as a plugin

...