Status

Current state: Under Discussion

Discussion thread: here

Vote thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The org.apache.kafka.common.utils package contains several utility classes that are used by Kafka internals but are also accessible from the Kafka clients artifact. This KIP focuses on three classes:

• org.apache.kafka.common.utils.Utils
• org.apache.kafka.common.utils.ByteBufferInputStream
• org.apache.kafka.common.utils.ByteBufferOutputStream

These classes are implementation helpers rather than APIs that Kafka intends to support as a long-term public contract. However, because they live in a non-internal package, external applications can import them directly.

This creates two problems. First, the package name makes these classes look like public APIs even though Kafka does not intend to provide a long-term compatibility guarantee for these helpers. Second, Kafka internals cannot safely evolve the implementation or remove methods without risking compatibility issues for applications that depend on these classes.

A GitHub Code Search survey on 2026-04-21 found direct public non-fork usage of all three classes. The counts below are GitHub Code Search result counts, not a complete census of unique downstream repositories, since private repositories and unindexed code are not visible.

This external usage is why this KIP proposes staged deprecation in Kafka 4.4.0 and removal in Kafka 5.0.0, rather than immediate removal in a minor release.

Public Interfaces

This KIP adds the following classes in Kafka 4.4.0:

• org.apache.kafka.common.utils.internals.Utils
• org.apache.kafka.common.utils.internals.ByteBufferInputStream
• org.apache.kafka.common.utils.internals.ByteBufferOutputStream

These classes provide the implementations used by Kafka internals.

They preserve the existing behavior of the corresponding classes in org.apache.kafka.common.utils. This KIP does not propose behavior changes.

Although these classes are public for Java accessibility reasons, they are located in an internals package and are not part of Kafka's public API compatibility contract.

This KIP deprecates the following existing classes in Kafka 4.4.0:

• org.apache.kafka.common.utils.Utils
• org.apache.kafka.common.utils.ByteBufferInputStream
• org.apache.kafka.common.utils.ByteBufferOutputStream

The old classes remain available with the same behavior during the Kafka 4.4.x line.

They will act as deprecated compatibility shims and will be removed in Kafka 5.0.0.

For example, the deprecated classes will look like the following compatibility shims. The examples are shortened to show the delegation pattern and do not list every public method.

For Utils, the shim extends the internal implementation so existing static method references remain source-compatible:

org.apache.kafka.common.utils.Utils:

@Deprecated(forRemoval = true, since = "4.4")
public final class Utils extends org.apache.kafka.common.utils.internals.Utils {
    private Utils() {}
}

org.apache.kafka.common.utils.ByteBufferInputStream:

@Deprecated(forRemoval = true, since = "4.4")
public final class ByteBufferInputStream extends InputStream {
    private final org.apache.kafka.common.utils.internals.ByteBufferInputStream delegate;

    public ByteBufferInputStream(ByteBuffer buffer) {
        this.delegate = new org.apache.kafka.common.utils.internals.ByteBufferInputStream(buffer);
    }

    @Override
    public int read() {
        return delegate.read();
    }

    @Override
    public int read(byte[] bytes, int off, int len) {
        return delegate.read(bytes, off, len);
    }

    @Override
    public int available() throws IOException {
        return delegate.available();
    }
}

org.apache.kafka.common.utils.ByteBufferOutputStream:

@Deprecated(forRemoval = true, since = "4.4")
public class ByteBufferOutputStream extends OutputStream {
    private final org.apache.kafka.common.utils.internals.ByteBufferOutputStream delegate;

    public ByteBufferOutputStream(ByteBuffer buffer) {
        this.delegate = new org.apache.kafka.common.utils.internals.ByteBufferOutputStream(buffer);
    }

    public ByteBufferOutputStream(int initialCapacity) {
        this.delegate = new org.apache.kafka.common.utils.internals.ByteBufferOutputStream(initialCapacity);
    }

    public ByteBufferOutputStream(int initialCapacity, boolean directBuffer) {
        this.delegate = new org.apache.kafka.common.utils.internals.ByteBufferOutputStream(initialCapacity, directBuffer);
    }

    @Override
    public void write(int b) {
        delegate.write(b);
    }

    @Override
    public void write(byte[] bytes, int off, int len) {
        delegate.write(bytes, off, len);
    }

    public void write(ByteBuffer sourceBuffer) {
        delegate.write(sourceBuffer);
    }

    public ByteBuffer buffer() {
        return delegate.buffer();
    }

    public int position() {
        return delegate.position();
    }

    public int remaining() {
        return delegate.remaining();
    }

    public int limit() {
        return delegate.limit();
    }

    public void position(int position) {
        delegate.position(position);
    }

    public int initialCapacity() {
        return delegate.initialCapacity();
    }

    public void ensureRemaining(int remainingBytesRequired) {
        delegate.ensureRemaining(remainingBytesRequired);
    }
}

These examples are illustrative. The implementation will preserve the existing public methods required for source and binary compatibility during the Kafka 4.4.x compatibility window.

Kafka's own codebase will migrate imports from org.apache.kafka.common.utils to org.apache.kafka.common.utils.internals for these three classes. This is an internal source-level migration only.

No Kafka protocol, configuration, wire format, metrics, command line tools, or client runtime behavior is changed by this KIP.

Proposed Changes

In Kafka 4.4.0:

1. Create org.apache.kafka.common.utils.internals.Utils as the implementation used by Kafka internals.
2. Create org.apache.kafka.common.utils.internals.ByteBufferInputStream as the implementation used by Kafka internals.
3. Create org.apache.kafka.common.utils.internals.ByteBufferOutputStream as the implementation used by Kafka internals.
4. Keep the existing classes in org.apache.kafka.common.utils in place, retaining only their public methods and delegating to the corresponding org.apache.kafka.common.utils.internals classes.
5. Mark the existing classes in org.apache.kafka.common.utils as deprecated for removal in Kafka 5.0.0.
6. Migrate Kafka's own source code to use the new org.apache.kafka.common.utils.internals classes.
7. Preserve the behavior of the old classes during the Kafka 4.4.x compatibility window.
8. Document the deprecation in the upgrade notes.

The new internal classes contain the original implementations moved from the existing classes. This KIP does not propose rewriting, redesigning, or changing the semantics of these classes.

During the Kafka 4.4.x release line, the deprecated public classes will delegate to the new internal classes, avoiding code duplication.

In Kafka 5.0.0:

1. Remove org.apache.kafka.common.utils.Utils.
2. Remove org.apache.kafka.common.utils.ByteBufferInputStream.
3. Remove org.apache.kafka.common.utils.ByteBufferOutputStream.
4. Keep the new org.apache.kafka.common.utils.internals classes for Kafka internal usage.

External applications should not migrate to the new org.apache.kafka.common.utils.internals classes. The new package is internal and may change without public API compatibility guarantees.

Compatibility, Deprecation, and Migration Plan

This KIP is source and binary compatible for Kafka 4.4.0 users because the existing classes in org.apache.kafka.common.utils remain available with the same behavior. The compatibility break is intentionally delayed until Kafka 5.0.0, where the deprecated classes will be removed.

The existing implementations are moved to org.apache.kafka.common.utils.internals, and the original classes delegate to them. Therefore, Kafka internal code should observe the same behavior after migrating imports to the internal package.

Kafka's own codebase will be migrated to use the new org.apache.kafka.common.utils.internals classes. This is an internal source-level migration and does not change Kafka client or broker runtime behavior.

Users compiling against Kafka 4.4.0 may see deprecation warnings if they import or reference the deprecated classes. These warnings indicate that the classes will be removed in Kafka 5.0.0.

External applications should not migrate to the new org.apache.kafka.common.utils.internals classes. The new classes are internal and may change without public API compatibility guarantees.

Applications that currently depend on these classes should replace that usage with JDK functionality, application-owned utility code, or a supported Kafka public API when one exists. If an application depends on a Kafka utility method or helper class whose behavior must remain stable for that application, the application should copy the needed logic into its own codebase and own that compatibility going forward.

The planned version behavior is:

Test Plan

The implementation will be tested by ensuring that all Kafka modules compile after internal usages are migrated to the new org.apache.kafka.common.utils.internals classes.

The existing tests for org.apache.kafka.common.utils.Utils, org.apache.kafka.common.utils.ByteBufferInputStream, and org.apache.kafka.common.utils.ByteBufferOutputStream will remain in place during the Kafka 4.4.x compatibility window. These tests should continue to pass after the classes are marked deprecated, verifying that the deprecated classes remain available and preserve existing behavior.

The internal implementations should also be covered by the same behavior expectations, either by moving the existing test coverage to the internal classes or by adding equivalent coverage for the internal package.

No new system test is expected because this KIP does not change runtime behavior, protocol behavior, configuration, wire format, metrics, or command line tools.

Rejected Alternatives

1. Move these classes directly to the internal package in Kafka 4.4.0.

This would remove the existing classes in org.apache.kafka.common.utils immediately. Although the classes were intended to be internal helpers, external usage exists, so immediate removal could break users during a minor release upgrade.

2. Keep the old classes indefinitely.

This would avoid compatibility risk, but it would continue to expose internal utility classes as if they were supported public APIs. Kafka would remain constrained by accidental external usage of helper classes that were not designed as public APIs.

3. Make selected helpers official public APIs.

Kafka could promote selected heavily used helpers to supported public APIs. This KIP does not propose that because it would expand Kafka's long-term compatibility surface and would require separate discussion about which utility methods or helper classes deserve a public contract.

If the community decides that a specific helper should become a supported public API, that should be handled in a separate KIP with a narrow API proposal.