Status

Current state: DraftVoting

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]thread: https://lists.apache.org/thread/hyxb9gwpsjdbxjs9rw8qsg6ws0nvlx0t

Voting thread: https://lists.apache.org/thread/5jl46khf1zvk6dcv37xj0bbd0tkg8bg3

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

With the adoption of JEP 411: Deprecate the SecurityManager for Removal in Java 17, the SecurityManager is now deprecated, and will be removed in an unknown future version.

In Java 23, the legacy APIs are now partially degraded, and throw UnsupportedOperationExceptions when a SecurityManager is not configured, so active progress is being made towards removal.

As Kafka supports multiple versions of Java (8, 11, 17, 21 for 3.x, 11, 17, 21 for 4.x), we should decide how to handle this deprecation and removal, and what replacements or mitigations we can provide for users of Kafka.

Kafka has a compile-time dependency on the AccessController  class in clients , core , and connect:runtime , and the removal of SecurityManager and it's accompanying classes would make these modules un-buildable with their current implementation.

Public Interfaces

The In a separate KIP, the Kafka project should to define a major release in which SecurityManager support will be removed. It is not possible to remove support for this feature in a minor release, as it would constitute a breaking change.will choose a major version to drop support for Java 17, and make the minimum supported version 21. In this same release, we will drop support for the SecurityManager entirely from Kafka.

From this major version forward, Kafka operators will not be able to secure their clients, brokers, or connector runtimes with the SecurityManager. As soon as this KIP is approved, Kafka users should consider the SecurityManager deprecated, and begin migrating away from it to prepare for the removal.

Proposed Changes

There are two strategies for approaching the removal that we should pursue concurrently:

...

For example, if SecurityManager is removed from the Kafka codebase in 4.0, and SecurityManager is removed from Java 25, users of Kafka 4.x can immediately upgrade to Java 25 without special considerations.

However, there are some problems with removing support quicklyconstraints on how early we can remove support:

1. Java 11 still has the non-deprecated SecurityManager, and Kafka 4.0 must support Java 11. If we were to remove SecurityManager support from Kafka 4.0, users on Java 11 would experience a removal without a prior deprecation.
2. Java 17 does not have the replacement Subject API https://bugs.openjdk.org/browse/JDK-8267108 , and Kafka 4.0 must support Java 17. We would need dual support for the old and new APIs.

Therefore, we will plan to remove SecurityManager support in the Kafka major version which drops support for JDK 17.

We can notify users of this removal with a notice in the Kafka documentation, and rely on the runtime's built-in deprecation warning log messagesThe earliest we could remove the SecurityManager without tech-debt is when Java 21 could become the minimum supported Java version. This is not planned for Kafka currently, but would probably be Kafka >=6.0.

Remove compile-time usages & detect SecurityManager removal at runtime

The longer it takes for Kafka to remove SecurityManager support, the more likely that the Java removal will take place before Kafka's removal is complete. To be prepared, we should anticipate that the project will need to support both versions with and without SecurityManager, using the same codebase & artifacts.

This can be done by replacing direct references to deprecated APIs with reflection, and gracefully changing the behavior for Java versions without the SecurityManager. When Java 21 becomes the minimum supported Java version, the reflective implementations could be removed and replaced with direct calls to the modern API.

Proposed Changes

The SaslClientCallbackHandler and OAuthBearerSaslClientCallbackHandler in clients  will no longer call Subject.getSubject , and instead will use the JDK 18+ Subject.current API: https://bugs.openjdk.org/browse/JDK-8267108 .
The SaslClientAuthenticator and SaslServerAuthenticator in clients will use the Java 18+ Subject.callAs API rather than the deprecated Subject.doAs function. This has a backwards-compatible implementation that will presumably be replaced when SecurityManager is removed.
The calls to getSubject, current, doAs, and callAs will be replaced with reflection. If Java 18 is available, the new functions will be used. Otherwise, the legacy API will be called.

The calls to AccessController.doPriviliged  in core  and connect:runtime: used for ClassLoader instantiation will be replaced with reflection. If the doPriviliged call is not available, the privileged operations will be done in the original context.

Compatibility, Deprecation, and Migration Plan

In an upcoming minor release after this KIP, the following deprecated calls will be changed to use reflection. If the legacy implementations are available and execute without throwing UnsupportedOperationException, they will be used. If not, the modern implementations will be called reflectively.

In the major release which removes SecurityManager support, the reflection will be replaced with direct calls to the modern implementations.

Compatibility, Deprecation, and Migration Plan

The static utility will be backwards-compatible, allowing the use of both the legacy SecurityManager bindings, or the modern replacements if the legacy bindings are degraded or removed.

Users upgrading to Java 17+ (in which SecurityManager was deprecated) will receive this warning message from the runtime, indicating they have a security manager dependency that will not be supported in the future. This will also serve as the runtime deprecation notice for Kafka.

WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future release

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by %s
WARNING: Please consider reporting this to the maintainers of %s
WARNING: System::setSecurityManager will be removed in a future release

Users upgrading to a version of Java without SecurityManager support will need to disable their security manager implementations/configurations entirely. Once the java.security.manager property is not specified, some or all of the legacy methods may throw UnsupportedOperationException, which will cause the new implementations to be used instead.

Removing SecurityManager support is a breaking change and will happen in a major version. Users upgrading to a version of Users upgrading to a version of Java or Kafka without SecurityManager support will need to disable their security manager implementations/configurations .Users upgrading to Java 17+ (in which SecurityManager was deprecated) already receive a warning message on startupentirely. This version will have no calls to the legacy methods at all, and will always use the new versions.

Test Plan

There are currently no tests exercising the SecurityManager interface in Kafka, and no new tests for the end-to-end behavior will be added for this project.

...

Since removing SecurityManager is a breaking change, we would have to either delay support for the new Java version significantly, or perform a major release of Kafka when we might not be prepared for it. These negatives outweigh the maintenance burden of making these function calls reflective.

Add the reflective utility, but follow the Java upstream's removal schedule

The reflective utility is a harmful piece of tech debt, and should be removed as early as possible. If we do not remove SecurityManager support in the Kafka project, we will be required to maintain the reflective utility until we no longer support JREs that have the SecurityManager.

For example, if Java 25 removes SecurityManager, we would need to maintain the reflective utility until Java 25 is the minimum supported version, which could be several years after Java 21 becomes the minimum supported version. If the upstream decides to remove SecurityManager support later, the Kafka project would be responsible for the tech debt for longer as well.

Removing our support for SecurityManager gives us control over the timeline with which to pay off this tech debt, and supports the Java community's goal of removing the functionality promptly.

Set a removal date for Java 11 / Java 17 support

This should be delegated to another KIP with a dedicated discussion. This KIP can be blocked on that one, or be conditional. For example, we can agree here that in the major release which removes Java 17 support, we will also remove SecurityManager support, but not specify when the removal of Java 17 support will take place, and the removal of SecurityManager support can be included in whatever version changes the minimum java version to 21.

Implement a alternative to the SecurityManager for use with Kafka

The SecurityManager is a flawed design, and we should not replicate it or try to solve those problems in a similar way. Instead we should encourage users to use the process boundary as a security boundary.