Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The org.apache.kafka.common.utils package contains several utility classes that are used by Kafka internals but are also accessible from the Kafka clients artifact. This KIP focuses on three classes:

• org.apache.kafka.common.utils.Utils
• org.apache.kafka.common.utils.ByteBufferInputStream
• org.apache.kafka.common.utils.ByteBufferOutputStream

These classes are implementation helpers rather than APIs that Kafka intends to support as a long-term public contract. However, because they live in a non-internal package, external applications can import them directly.

This creates two problems. First, the package name makes these classes look like public APIs even though Kafka does not intend to provide a long-term compatibility guarantee for these helpers. Second, Kafka internals cannot safely evolve the implementation or remove methods without risking compatibility issues for applications that depend on these classes.

A GitHub Code Search survey on 2026-04-21 found direct public non-fork usage of all three classes. The counts below are GitHub Code Search result counts, not a complete census of unique downstream repositories, since private repositories and unindexed code are not visible.

This external usage is why this KIP proposes staged deprecation in Kafka 4.4.0 and removal in Kafka 5.0.0, rather than immediate removal in a minor release.

Public Interfaces

This KIP adds the following classes in Kafka 4.4.0:

• org.apache.kafka.common.utils.internals.Utils
• org.apache.kafka.common.utils.internals.ByteBufferInputStream
• org.apache.kafka.common.utils.internals.ByteBufferOutputStream

These classes are created by copying the existing implementations from org.apache.kafka.common.utils into the org.apache.kafka.common.utils.internals package. They are not new implementations and this KIP does not propose behavior changes.

Although these classes are public for Java accessibility reasons, they are located in an internals package and are not part of Kafka's public API compatibility contract.

This KIP deprecates the following existing classes in Kafka 4.4.0:

• org.apache.kafka.common.utils.Utils
• org.apache.kafka.common.utils.ByteBufferInputStream
• org.apache.kafka.common.utils.ByteBufferOutputStream

The old classes remain available with the same behavior during the Kafka 4.4.x line and will be removed in Kafka 5.0.0.

No Kafka protocol, configuration, wire format, metrics, command line tools, or client behavior is changed by this KIP.

Proposed Changes

In Kafka 4.4.0:

1. Copy the existing org.apache.kafka.common.utils.Utils implementation to org.apache.kafka.common.utils.internals.Utils.
2. Copy the existing org.apache.kafka.common.utils.ByteBufferInputStream implementation to org.apache.kafka.common.utils.internals.ByteBufferInputStream.
3. Copy the existing org.apache.kafka.common.utils.ByteBufferOutputStream implementation to org.apache.kafka.common.utils.internals.ByteBufferOutputStream.
4. Keep the existing classes in org.apache.kafka.common.utils in place.
5. Mark the existing classes in org.apache.kafka.common.utils as deprecated for removal in Kafka 5.0.0.
6. Migrate Kafka's own source code to use the new org.apache.kafka.common.utils.internals classes.
7. Preserve the behavior of the old classes during the Kafka 4.4.x compatibility window.
8. Document the deprecation in the upgrade notes.

The new internal classes are exact copies of the existing implementations at the time of migration. This KIP does not propose rewriting, redesigning, or changing the semantics of these classes.

The old and new classes will intentionally duplicate the implementation during the Kafka 4.4.x release line. This duplication is temporary compatibility debt. It avoids large hand-written forwarding classes and avoids changing the shape of the utility classes only to support the migration.

In Kafka 5.0.0:

1. Remove org.apache.kafka.common.utils.Utils.
2. Remove org.apache.kafka.common.utils.ByteBufferInputStream.
3. Remove org.apache.kafka.common.utils.ByteBufferOutputStream.
4. Keep the new org.apache.kafka.common.utils.internals classes for Kafka internal usage.

External applications should not migrate to the new org.apache.kafka.common.utils.internals classes. The new package is internal and may change without public API compatibility guarantees.

Compatibility, Deprecation, and Migration Plan

This KIP is source and binary compatible for Kafka 4.4.0 users because the existing classes in org.apache.kafka.common.utils remain available with the same behavior. The compatibility break is intentionally delayed until Kafka 5.0.0, where the deprecated classes will be removed.

The newly added org.apache.kafka.common.utils.internals classes are copied from the existing org.apache.kafka.common.utils implementations. Therefore, Kafka internal code should observe the same behavior after migrating imports to the internal package.

Users compiling against Kafka 4.4.0 may see deprecation warnings if they import or reference the deprecated classes. These warnings indicate that the classes will be removed in Kafka 5.0.0.

External applications should not migrate to the new org.apache.kafka.common.utils.internals classes. The new classes are internal and may change without public API compatibility guarantees.

Applications that currently depend on these classes should replace that usage with JDK functionality, application-owned utility code, or a supported Kafka public API when one exists. If an application depends on a Kafka utility method or helper class whose behavior must remain stable for that application, the application should copy the needed logic into its own codebase and own that compatibility going forward.

The planned version behavior is:

Test Plan

The implementation will be tested by ensuring that all Kafka modules compile after internal usages are migrated to the new org.apache.kafka.common.utils.internals classes.

The existing tests for org.apache.kafka.common.utils.Utils, org.apache.kafka.common.utils.ByteBufferInputStream, and org.apache.kafka.common.utils.ByteBufferOutputStream will remain in place during the Kafka 4.4.x compatibility window. These tests should continue to pass after the classes are marked deprecated, verifying that the deprecated classes remain available and preserve existing behavior.

The copied internal implementations should also be covered by the same behavior expectations, either by moving the existing test coverage to the internal classes or by adding equivalent coverage for the internal package.

No new system test is expected because this KIP does not change runtime behavior, protocol behavior, configuration, wire format, metrics, or command line tools.

Rejected Alternatives

1. Move these classes directly to the internal package in Kafka 4.4.0.

This would remove the existing classes in org.apache.kafka.common.utils immediately. Although the classes were intended to be internal helpers, external usage exists, so immediate removal could break users during a minor release upgrade.

2. Keep the old classes indefinitely.

This would avoid compatibility risk, but it would continue to expose internal utility classes as if they were supported public APIs. Kafka would remain constrained by accidental external usage of helper classes that were not designed as public APIs.

3. Make selected helpers official public APIs.

Kafka could promote selected heavily used helpers to supported public APIs. This KIP does not propose that because it would expand Kafka's long-term compatibility surface and would require separate discussion about which utility methods or helper classes deserve a public contract.

If the community decides that a specific helper should become a supported public API, that should be handled in a separate KIP with a narrow API proposal.