Status

Current state: Discussion
Authors: Eric Chang
Discussion thread: https://lists.apache.org/thread/5hf11khclhngftp297hhvwc23j5bcx28
Previous vote thread (earlier proposal): https://lists.apache.org/thread/qx1y2nvx27h1v1t0zvp9kqolnd3fhxls
JIRA: https://issues.apache.org/jira/browse/KAFKA-20411

Please keep the discussion on the mailing list rather than commenting on the wiki.

Motivation

Kafka's published Java archives contain classes that applications can access even though Kafka does not intend to support them as Public APIs. Some are implementation helpers placed outside packages named internal or internals, so external Java and Scala applications can import them directly.

KIP-1265 defines a class as effectively Public when @InterfaceAudience.Public applies either directly or through an enclosing Public class. A direct @InterfaceAudience.Private annotation overrides that inherited coverage. Classes without effective Public coverage are Private. KIP-1265 also provides a checker for compiled classes that external projects can use to find internal API usage. This gives Kafka a consistent default, but it does not answer every compatibility question.

In this KIP, a Public contract means an API or extension point that Kafka intends external applications to depend on and maintainers to support across releases.

There are two important cases among classes that are not effectively Public:

1. Some are exposed through an existing Public API, implement a Public Kafka type, appear in Public Javadoc, or are used through a documented extension mechanism. These may indicate a missing Public annotation or another API-definition issue.
2. Others are Kafka implementation details that downstream source code imports directly. These remain Private, but immediate removal can have a large compatibility impact when usage is widespread.

The KIP-1265 annotation alone cannot distinguish these cases. It also does not produce a Java compiler deprecation warning or define a removal schedule for a heavily used internal class.

KIP-1320 therefore proposes staged deprecation only for classes that Kafka maintainers confirm are internal and whose external usage justifies a warning period. Staged deprecation keeps the old class available with a compiler warning for a stated release period before removal. Each proposed class has migration guidance and a stated removal release. The initial proposed set contains only org.apache.kafka.common.utils.Utils.

The class scan and external-usage count support that decision; they do not make it. This KIP does not create a compatibility guarantee for every class classified as Private by KIP-1265. A class receives a warning period only when maintainers approve it in the Deprecation Registry, the table of classes selected for staged deprecation.

Candidate Identification

In this section, a candidate is an externally accessible class that KIP-1265 classifies as Private. An active candidate is a candidate that is not already deprecated and is not already under an internal or internals package. The following flow shows how the scan narrows the classes that maintainers review:

+------------------------------------------------------------------+
| One fixed Kafka source revision + ten published client modules   |
+------------------------------------------------------------------+
                                |
                                v
+------------------------------------------------------------------+
| Scan compiled class files for externally accessible classes      |
+------------------------------------------------------------------+
                                |
                                v
+------------------------------------------------------------------+
| Apply the KIP-1265 effective-audience rules                      |
+------------------------------------------------------------------+
       |
       +-- Public --> outside scope
       |
       +-- Private --> 3,226 candidates
                                |
                                v
+------------------------------------------------------------------+
| Exclude classes already deprecated or in internal/internals      |
+------------------------------------------------------------------+
       |
       +-- excluded --> already handled; do not search for usage
       |
       +-- remaining --> 2,227 active candidates
                              |
                 +------------+------------+
                 |                         |
                 v                         v
+--------------------------------+  +------------------------------+
| Check whether a supported API  |  | Count matching Java + Scala  |
| already exposes or uses the    |  | import files on GitHub for   |
| candidate intentionally        |  | every active candidate       |
+----------------+---------------+  +---------------+--------------+
                 |                                  |
                 +----------------+-----------------+
                                  |
                                  v
+------------------------------------------------------------------+
| Match both records by fully qualified class name (FQCN)          |
+------------------------------------------------------------------+
                                  |
                                  v
+------------------------------------------------------------------+
| Combined Java + Scala matching files >= 1,000?                   |
+------------------------------------------------------------------+
       |
       +-- no --> keep for later review
       |              |
       |              +-- maintainer nomination ----+
       |                                            |
       +-- yes --> prioritize 75 classes -----------+
                                                    |
                                  +-----------------+
                                  |
                                  v
+------------------------------------------------------------------+
| Maintainer assigns a class-specific outcome                      |
+------------------------------------------------------------------+

The two checks answer different questions. The first check asks whether Kafka already treats the class as supported, even though its annotation says Private. The GitHub search asks a different question: how much external source code may be affected if maintainers confirm that the class is internal. Neither result determines the outcome by itself.

How the results are combined. Each class is identified by its FQCN, for example org.apache.kafka.common.utils.Utils. The scan keeps every active candidate and attaches the Java and Scala usage counts that match that name. The combined usage count is the sum of those two matching-source-file counts; it is not a count of unique users, projects, or repositories. A class does not need a positive result from both checks to remain in the data. The 1,000-file threshold prioritizes 75 classes for review; it does not classify them or prevent maintainers from nominating another class.

Maintainers use one of these outcomes after reviewing a class. The counts describe the current author proposals, not community agreement:

Maintainers interpret the combined record in this order:

1. Does the first check show that the class is already part of a supported API, extension, plugin, or configuration entry point?
2. If not, do maintainers confirm that the class is an internal implementation detail?
3. If it is internal, does the observed external impact justify a warning period, or is a direct internal change sufficient?
4. Is the class already handled by another Jira, PR, or KIP?

Examples from the current author proposals are shown below:

Deprecation Strategy

A candidate receives staged deprecation only when maintainers make both of these decisions:

1. The class is an internal implementation detail rather than an existing Public API or supported extension, plugin, or configuration entry point.
2. Its external compatibility impact justifies a warning period instead of a direct internal change.

For every accepted staged-deprecation candidate, the Deprecation Registry records:

• the release that adds @Deprecated(since = "...", forRemoval = true);
• a supported replacement, application-owned alternative, or an explicit statement that no replacement exists;
• the compatibility mechanism used during the warning period;
• the removal release and tracking work.

Kafka migrates its own callers away from the deprecated class. If the implementation must move so Kafka can continue compiling without internal deprecation warnings, Kafka may keep a compatibility shim: the old class remains temporarily and forwards calls to the new internal implementation. This forwarding form is also called a delegating shim. A shim is selected per class; it is not required for every staged deprecation, and it must not duplicate implementation logic.

The compatibility surface is the set of old class names, fields, and methods that remain available during that warning period.

The automated observations can be discussed in groups, but the final outcome remains a per-class maintainer decision recorded in Appendix B. Scan results, usage counts, and threshold membership never select staged deprecation automatically.

Deprecation Registry

Only classes explicitly approved for staged deprecation appear here. Candidate-list membership, scan observations, and pending review do not create a compatibility guarantee or authorize removal.

This draft intentionally does not decide the process for adding a future class to this registry after KIP-1320 is approved. Until that open question is resolved, this proposal authorizes only the rows agreed as part of the KIP vote; currently that is Utils.

Public Interfaces

The class scan and discussion process do not create a Public API.

For each class selected for a code change, this section must record the affected class, its intended treatment, and its compatibility boundary. The current concrete proposal is:

The current Public-interface change is the Utils staged transition. Kafka 4.5 retains org.apache.kafka.common.utils.Utils as a deprecated compatibility shim while Kafka's implementation moves to org.apache.kafka.common.utils.internals.Utils; Kafka 5.0 removes the old shim.

The new internals.Utils class is public only because Java requires that visibility for Kafka callers in other packages. It is not added to Kafka's Public API contract. Downstream users must migrate to standard Java functionality, application-owned code, or an existing supported Kafka API rather than importing the internal implementation.

Appendix C records Utils as a worked example of the delegating-shim strategy, including its usage evidence, forwarding pattern, compatibility details, and release behavior. The shim pattern is not automatically selected for another candidate.

Maintainer Review

The six outcomes and their current counts are defined in Candidate Identification, where they first appear. The scan supplies evidence for review, but high external usage alone does not select an outcome. Only Utils is currently proposed for staged deprecation by this KIP.

For every class explicitly accepted for staged deprecation, the change must:

1. identify the affected class;
2. keep the old class as a deprecated compatibility shim when compatibility is required;
3. document the major-release removal boundary;
4. move Kafka's own callers to a non-deprecated internal implementation;
5. preserve the required source and binary compatibility surface during the transition;
6. add targeted compatibility tests and upgrade notes.

Normal Jira tracking, code review, testing, and upgrade-note requirements still apply. Promoting a class to the supported Public API surface is a separate decision and may require a separate KIP.

Whether a future staged-deprecation row can be added through a KIP-1320 amendment or requires a new KIP and vote remains an open question. This draft grants no such future authorization.

Proposed Changes

For every class accepted into the Deprecation Registry, Kafka will:

1. add @Deprecated with the recorded since and forRemoval values;
2. publish migration guidance and the removal release;
3. migrate Kafka's own callers away from the deprecated symbol;
4. preserve only the compatibility surface recorded for that class;
5. add targeted compatibility tests and an upgrade-note entry.

The current registry applies this strategy to Utils. Kafka 4.5 moves its implementation to internals.Utils, retains the old class as a deprecated delegating shim, and migrates Kafka's Java and Scala callers. Kafka 5.0 removes the shim. Appendix C explains why this class needs the shim variant and how it preserves compatibility without duplicating implementation logic.

Compatibility, Deprecation, and Migration Plan

The class scan itself does not change compatibility.

Classes classified as Private by KIP-1265 do not automatically receive a new guarantee from KIP-1320. Only a class explicitly assigned staged deprecation receives the documented compatibility period. A class assigned a direct internal move remains outside Kafka's Public API compatibility contract.

During a staged-deprecation warning period, existing source and binary symbols remain available, but downstream builds that treat deprecation warnings as errors may need to migrate or suppress the warning. The recorded removal release is the intentional compatibility boundary.

For the current Utils row, Kafka 4.5 preserves the old compatibility surface and Kafka 5.0 removes it. Appendix C records the class-specific migration guidance, version behavior, and compatibility properties.

Test Plan

The identification tooling records the Kafka source revision and content fingerprints for itself, its dependencies, the scanned Java archives, and the generated list of usage searches. A reproducibility test runs the local class scan twice and verifies byte-identical outputs. The usage search stores Java and Scala counts separately, then computes the combined usage count from them.

Each class selected for a code change requires tests appropriate to its recorded outcome.

Every staged-deprecation change must verify the annotation, Kafka caller migration, documented compatibility surface, migration guidance, and removal metadata. A compatibility shim additionally requires tests that it exposes the required old fields and methods and forwards calls correctly.

Appendix C lists the Utils-specific compatibility tests. No new system test is expected for that example because it does not change runtime behavior, protocol behavior, configuration, wire format, metrics, or command-line tools.

Appendix A. Identification Methodology

The accompanying proof of concept (PoC) scans compiled class files from one fixed Kafka source revision. Its current scope contains ten published modules commonly used by external applications:

• kafka-clients
• kafka-streams
• kafka-streams-test-utils
• connect-api
• connect-transforms
• connect-json
• connect-file
• connect-mirror-client
• connect-mirror
• connect-basic-auth-extension

The PoC scans every externally accessible class in these modules, but it does not cover every Kafka Gradle module.

This scan produces the class inventory: one record for every externally accessible class that is not effectively Public under the KIP-1265 model. Classes already deprecated or already under an internal or internals package remain recorded but are excluded from usage searches. The query manifest is the generated list of searches to run for the remaining active candidates.

For each class, the PoC records observable characteristics that may affect a maintainer's decision:

• direct or indirect appearance in a supported field or method definition;
• inheritance from, or implementation of, an effectively Public Kafka type;
• a reference from Kafka configuration or service-provider metadata, meaning files that name plugin implementations;
• appearance in Public Javadoc without effective Public coverage;
• existing deprecation or placement under an internal package.

These characteristics can overlap. They are evidence for discussion and do not automatically decide whether a class is Public or internal.

Here, a Public signature means the types exposed by a public or protected field or method, including generic types, declared exceptions, and inherited types. The scanner begins with effectively Public classes and records whether it reaches a candidate directly or indirectly through another type. Configuration evidence is recorded separately and does not count toward the external source-usage threshold.

The generated files preserve these observations as evidence tags. Appendix B displays the exact tags, defined here:

The PoC uses GitHub Code Search to run the same two source queries for every active candidate. The {FQCN} placeholder is the fully qualified class name defined above.

"import {FQCN};" language:Java NOT repo:apache/kafka NOT is:fork
"import {FQCN}" language:Scala NOT repo:apache/kafka NOT is:fork

Configuration-only references are not counted. For example, configuring org.apache.kafka.connect.json.JsonConverter as a Connect converter is an intended plugin mechanism rather than an accidental Java source dependency.

Java and Scala counts are recorded separately. Their sum is the combined usage count used to prioritize discussion. It counts matching source files, not unique projects or users.

The deprecation itself is not Java-only. java.lang.Deprecated is recorded in the compiled class and is visible to Scala callers. Measuring only Java imports would still undercount the compatibility impact, so the usage survey includes Scala source separately. Other languages that compile for the Java Virtual Machine are outside this first query scope and can be added as versioned queries.

The usage snapshot is the saved set of search results and the information needed to trace how they were produced. The completed snapshot, named JVM_SOURCE_IMPORT_TEXT_V1, is anchored to Kafka commit ac0d7e245b213c20ab66bcea211d1cbd5ed2c95a. It contains 2,227 active candidates and 4,454 query records, where one record stores the query and result for one class and one language. Seventy-five classes had at least 1,000 combined Java and Scala matching files. The observations for those 75 candidates are:

The rows can overlap and must not be added together. The 1,000-file threshold only reduces the number of classes that maintainers need to discuss. It is not a compatibility or deprecation policy.

The final usage snapshot combines completed results from an earlier Java search and a later Scala search rather than repeating unchanged queries. A saved result is reused only when the query text, class, language, and Kafka source revision match exactly. It reused 2,227 Java records and 2,223 Scala records. Only four Scala records missing from the later search—Utils and its three nested types—were queried again. The saved metadata retains the original result, time, and content fingerprints so the origin of every record remains traceable. The query window, from the earliest reused result to the four new searches, is 2026-07-19T01:11:20Z to 2026-07-21T01:03:13Z; 4 records were newly queried and 4,450 were reused. The branch includes the generated 75-row candidate result and its 150 Java and Scala GitHub query rows, so reviewers can inspect the selected evidence without rerunning the crawl.

The text queries have known limits. The Scala query uses prefix matching, so an outer-class query can also match imports of its nested types. It does not find grouped imports, renamed imports, package wildcard imports, or references that spell out the full package and class name without an import. The Java query does not find static imports, package wildcard imports, or those fully spelled-out references. Neither query includes private repositories or unindexed code. The query asks GitHub to exclude forks with NOT is:fork, but copied or mirrored source can remain. These results are reproducible usage signals, not a complete census of Java, Scala, or other code targeting the Java runtime.

The PoC uses GitHub's web API for Code Search. GitHub reported that this interface was deprecated on 2026-03-27 and is scheduled to stop operating on 2026-09-27. The snapshot saves that notice with the results so a replacement search method can be introduced as a new version without silently changing how usage is measured.

Appendix B. Candidate Review Registry

The following table is the decision registry for the 75 candidates in the completed JVM_SOURCE_IMPORT_TEXT_V1 snapshot. It is sorted by the combined Java and Scala matching-file count. The evidence tags are observations from the scanner, not API classifications. A Pending maintainer review row creates no compatibility guarantee and authorizes no code change.

The six outcome values are: Staged deprecation, Public API follow-up, Intended entry point, Direct internal change, Handled elsewhere or no change, and Pending maintainer review. A non-pending outcome must identify its transition or replacement, release boundary when applicable, and tracking work. A pending row identifies a question category rather than presenting a decision. Status distinguishes an author proposal from community agreement and implementation.

Pending rows use these question categories:

The threshold is a reproducible way to bound the initial review, not a rule that prevents maintainers from nominating another class. Maintainer-nominated exceptions are tracked separately so they do not silently change the 75-class threshold result:

The outcome-count summary in Candidate Identification covers only the reproducible 75-class threshold set. Maintainer nominations are reported separately and follow the same review process.

Author proposals for the strongest Public-contract signals (local only; not yet community-agreed):

• Schema: Direct Public-signature exposure through @InterfaceAudience.Public MirrorMaker Checkpoint and Heartbeat schema fields. Those fields already carry @SuppressKafkaInternalApiUsage, which records an intentional internal-API reference and notes that a KIP must promote the type or refactor the Mirror surface. Related protocol types (Field, Struct, Type) share that Mirror path and remain pending until the Mirror decision is settled.
• JsonConverter: Implements the Public Converter and HeaderConverter interfaces, is referenced from Connect example configs and META-INF/services, and matches the role of already-annotated Public StringConverter. Configuration-only use remains intentional; the open issue is the missing Public annotation on the concrete class.
• SimpleConfig: Extends Public AbstractConfig and is widely imported by third-party SMTs, but superclass Public status alone does not make the subclass effectively Public. The author proposal is to decide whether Kafka intends a Public SMT helper here, or whether callers should use AbstractConfig directly.
• RequestContext: Implements Public AuthorizableRequestContext, which is the supported authorizer plugin contract. The concrete class remains broker plumbing; external callers should depend on the interface.

Appendix C. Worked Example: Utils

Utils demonstrates the staged-deprecation strategy when Kafka must retain a widely used class while moving its implementation out of the deprecated class. The details in this appendix are required for the Utils registry row. Calling it an example means only that this transition is not the default for every other candidate.

Why staged deprecation was selected

• The scanner found no existing Public-contract signal for org.apache.kafka.common.utils.Utils.
• The completed usage snapshot found 2,284 Java and 1,304 Scala matching files, for a combined count of 3,588.
• The author proposes to treat the class as an internal utility rather than a supported long-term Public API.
• The author judges the observed compatibility impact sufficient to justify a warning period instead of an immediate minor-release removal.

Why this example uses a shim

Kafka's own Java main source compiles with deprecation warnings treated as errors. Deprecating Utils in place while leaving Kafka's internal callers unchanged would either fail the build or require broad suppressions. Kafka therefore moves the implementation to org.apache.kafka.common.utils.internals.Utils, migrates its own callers, and retains the old class only as a deprecated delegating compatibility surface.

The implementation logic exists only in the internal class. The following shortened example shows the pattern and intentionally omits most forwarding methods:

@Deprecated(since = "4.5", forRemoval = true)
public final class Utils {
    private Utils() {}

    public static final String NL =
        org.apache.kafka.common.utils.internals.Utils.NL;

    public static int abs(int value) {
        return org.apache.kafka.common.utils.internals.Utils.abs(value);
    }

    @FunctionalInterface
    public interface SwallowAction
            extends org.apache.kafka.common.utils.internals.Utils.SwallowAction {
    }

    public static void swallow(SwallowAction action) {
        org.apache.kafka.common.utils.internals.Utils.swallow(action);
    }
}

The complete shim forwards every existing public static method and field. Explicit forwarding is required because Scala does not expose inherited Java static methods in the same way as Java. The shim also preserves the names used by compiled code to link the nested interfaces, the existing logger name org.apache.kafka.common.utils.Utils, and the existing fallback behavior that tries Kafka's own class loader when the default loader cannot find a class.

No replacement Public API is proposed. Downstream users should use standard Java functionality, application-owned utility code, or an existing supported Kafka Public API. They must not migrate to org.apache.kafka.common.utils.internals.Utils.

Version behavior

Example-specific tests

• Kafka Java and Scala main sources compile after caller migration.
• Java main source compiles without deprecation warnings under -Xlint:all -Werror.
• Existing behavior tests run against the internal implementation.
• Compatibility tests verify the annotation, public fields, every public static forwarding method, and nested-interface names, and confirm that the shim exposes the required old surface.
• Tests verify the existing logger name and fallback Kafka class-loader behavior.

No Kafka protocol, configuration, wire format, metrics, command-line, or runtime behavior changes as part of this example.

Alternatives considered for this example

• Direct move in 4.5: rejected because it would remove a heavily imported symbol in a minor release.
• Keep Utils indefinitely: rejected because it would preserve an accidental API constraint without making the class a supported Public API.
• Promote selected helpers: rejected because each helper would require a separate Public API design decision.
• Deprecate in place without moving the implementation: rejected because Kafka's internal callers compile with deprecation warnings treated as errors; broad suppressions would be required.

Rejected Alternatives

1. Treat every class without @InterfaceAudience.Public as the same kind of internal API.

   This would ignore Public signature exposure, Public supertypes, documented plugin mechanisms, and possible annotation gaps. The KIP-1265 default is the starting point, not enough evidence for the final transition decision.

2. Deprecate every heavily used class that is effectively Private under KIP-1265.

   High external usage does not prove that every class should receive the same treatment. Some classes may be missing a Public annotation because they are exposed through an existing Public API. Others are implementation helpers, plugin implementations, generated protocol types, or classes already covered by separate work. Applying deprecation without reviewing these differences would create unnecessary maintenance work and could send incorrect guidance to users.

3. Treat a high external import count as proof that a class should become Public.

   Usage shows compatibility impact, not API quality or Kafka's intent to maintain the class as a long-term contract. A Public API decision must consider semantics, ownership, replacement guidance, and future compatibility cost.

4. Rely only on the KIP-1265 Private classification and checker.

   The Private classification does not produce a Java compiler warning, and the checker only helps projects that install it. It also does not define a transition schedule for heavily used internal classes.