Current state: Accepted
Discussion thread: here
Connect Framework offers REST API that is used to mange the lifecycle of the connector. Its imperative in most enterprises to secure the API and also add authorization to the end points. We could add the ability for authentication and authorization in the framework. But the security requirements are so broad that it's not practical to support all of them in the framework. Hence we must provide ability for users to plug resources that help achieve the required capabilities.
While security is prime use cases for this extension. Its not limited to that. Some of the common use cases are
- Build a custom Authentication filter
- Build a custom Authorization filter
- Complex extensions can even provide filters that rewrite/validate the connector requests to enforce additional constraints on the connector configurations
Developers would be required to implement only the ConnectRestExtension interface to provide an extension. ConnectRestExtension provides an implementation of ConnectRestExtensionContext whose configurable() provides an implementation of javax.ws.rs.core.Configurable. Using this developers can register new JAX-RS resources like filter, new end points, etc
As mentioned above, even though the developers are required to only implement the ConnectRestExtension, they will be using several new public interfaces that are implemented by the framework.
A new Versioned interface that will be used by all the plugins/components that support version. The Connector interface would be modified to extend this new interface instead of exposing the version() method itself.
This is a request Context interface that composes and provides access to
- Configurable - register JAX-RS resources
- clusterState - A new interface that helps provide some cluster state information
This interface provides methods for the extension to get the connector states and list of running connectors.
This also introduces a new configuration that rest.extension.classes that allows to configure a comma separated list of Rest extension implementations.
Users will be able to create a plugin by implementing the ConnectRestExtension interface, which has a single method that takes a ConnectRestExtensionContext instance as the only parameter. This allows us to change the interface easily in future to add new parameters. Connect runtime would also provide a default implementation for the interface ConnectRestExtensionContext. One or more of the ConnectRestExtension implementation can be configured via the configuration rest.extension.classes as a comma separated list of class names.
Implementations would use the
javax.ws.rs.core.Configurable to register one or more JAX-RS resources and get access to the Worker's Configs through the configure(Map<String, ?> configs) method in the ConnectRestExtension implementation( through org.apache.kafka.common.Configurable)
We will be introducing another new public API ConnectClusterState which will at present provide some of the read only methods from the Herder. The change would also include a default implementation ConnectClusterStateImpl in the connect runtime that will delegate to the underlying Herder. This will be useful when you want to add new resources like healthcheck, monitoring, etc.
Rest Extension Integration with Connect
The plugin's would be registered in the RestServer.start(Herder herder) method after registering the default Connect resources. Connect Runtime would provide an implementation of Configurable interface that would do the following.
- Constructed with the ResourceConfig available in the RestServer and the configure(Map<String, ?> configs) is invoked on the implementation
- Will check if resource is already registered. If not, it would delegate to ResourceConfig. If already registered would log a warning message.
- For non-register methods would just delegate to the ResourceConfig instance. This helps alleviate any issues that could arise if Extension accidentally reregister the connect resources.
- The close() for the plugins would be invoked as part of the stop() in the RestServer. The implementation would not be invoked after this.
The new extension class and its dependencies would need to be as part of the plugin path. Hence ConnectRestExtension would be defined as a new plugin to be loaded by the PluginClassLoader.The plugin would be looked up based on Java's Service Provider API instead of the Reflections scan that is used for other plugins. This will help in terms of not adding class loader cost that is associated in scanning the classes today for other plugins. Hence the implementation must provide a `META-INF/services/org.apache.kafka.connect.rest.ConnectRestExtension` as part of the jar file containing the fully qualified implementation class .
Consider the following example that defines a single plugin to add an authenticating filter and a health check resource.
For the RestExtension implementation to be found, the JAR should include the classes required by the implementation (excluding any Connect API or JAX-RS JARs) and should include a META-INF/services/org.apache.kafka.connect.rest.ConnectRestExtension file that contains the fully-qualified names of the extension implementation class(es) found within the JAR. This is the standard Java Service Loader API mechanism.
The above illustrated plugin can then be configured in the worker's configuration as below
The KIP proposes to include a reference implementation that allows users to authenticate incoming Basic Auth headers against the configured JAASLoginModule.
Compatibility, Deprecation, and Migration Plan
- This is entirely new functionality so there are no compatibility, deprecation, or migration concerns.
- Creating configs specific to the plugin and just passing them to the plugin based on a prefix. It was considered much easier to make the complete WorkerConfig. Also, in many cases the plugins would need to know just more than their configs to implement their actions.
- Passing the Herder to the plugin was considered but it was rejected since the Herder API is not public and we don't want to expose the complete Herder capabilities to the plugin.
- Providing ability to just add Filters insteda of any kind of Jersey resource was considered but it was rejected because it was too limiting in its capability that one canot add new resource end points or add a jersey provider.
- Having the Connect REST plugin as part of class path is rejected for the same reason why we have custom interfaces like Converters and Connectors as plugin and loaded via plugin path.