Status

Current state: Approved

Discussion thread: thread-1 and thread-2

Voting: thread 3 binding votes

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Apache Kafka currently specifies that any class which publishes Javadoc is Public — see Kafka Improvement Proposals#What is considered a "major change" that needs a KIP. To designate an interface as a public API, implementers must additionally define a Gradle rule following the pattern established in build.gradle.

However, there is a risk that builders may inadvertently import / expose internal classes — see KIP-1247: Make Bytes utils class part of the public API.

This KIP proposes to introduce a mechanism that:

  • Explicitly declares the audience of every public class.
  • Restricts the unintentional exposure of internal interfaces.
  • Enables automated detection of internal API usage for plugin developers, including Java, Scala, and Kotlin consumers.
  • Provides an auditable escape hatch for known/intentional violations.

Public Interfaces

Two new annotations are introduced in org.apache.kafka.common.annotation, modelled on Hadoop's @InterfaceAudience and mirroring the shape of the existing InterfaceStability class (outer class + nested annotations). The audience and stability dimensions are orthogonal — a class may carry both an audience annotation and an InterfaceStability annotation.

@InterfaceAudience


package org.apache.kafka.common.annotation;

public class InterfaceAudience {
    /** Intended for end users of Apache Kafka. */
    @Documented
    @Retention(RetentionPolicy.RUNTIME)
    public @interface Public { }

    /** Intended for internal use within Apache Kafka. No compatibility guarantees. */
    @Documented
    @Retention(RetentionPolicy.RUNTIME)
    public @interface Private { }
}

Default: if no audience annotation is present on a class, it is treated as @InterfaceAudience.Private. External code must not depend on classes without an explicit @InterfaceAudience.Public annotation.

Example combining audience and stability:


@InterfaceAudience.Public
@InterfaceStability.Evolving
public class AlterClientQuotasResult {
    // public API whose interface might change in the future
}

Design note: A LimitedPrivate audience was considered (per Hadoop) for plugin/connector-developer surfaces. It was rejected because (a) externally we cannot verify the caller's audience without intrusive build-config, and (b) InterfaceStability already carries the contract-narrowing signal. Two values — Public and Private — keep the model simple.

@SuppressKafkaInternalApiUsage

A new public annotation that grants an auditable opt-out for known/intentional references to internal Kafka classes from consumer code. Required for projects that wish to adopt the checker but already have legitimate exposures.


package org.apache.kafka.common.annotation;

@Documented
@InterfaceAudience.Public
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD, ElementType.FIELD, ElementType.CONSTRUCTOR})
public @interface SuppressKafkaInternalApiUsage {
    /** Human-readable justification; printed in the checker's report. */
    String value() default "";
}

Scope rules:

  • A class-level annotation suppresses every reference in the class, including references made from synthetic lambda methods (lambda$<m>$N).
  • A method/field/constructor-level annotation suppresses only references inside that member (and its header).
  • An empty value() is allowed but rendered as (no reason given) in the report.

Each suppressed reference is recorded in the build report with its location and the value() reason, so reviewers can audit every opt-out.

Proposed Changes

Annotating Apache Kafka code

A migration script will identify every class with Javadoc (today's implicit public-API marker) and add @InterfaceAudience.Public. @InterfaceAudience.Private will be added to classes that publish Javadoc only by accident. The audit will respect existing @InterfaceStability annotations and leave them untouched.

Restrictions enforced by the build

  • Public APIs must not expose non-public types via their public method signatures, return types, inner classes, or field types.
  • A class marked @InterfaceAudience.Public must appear in the Javadoc jar; a class in the Javadoc jar must carry @InterfaceAudience.Public. Either mismatch fails the build.
  • These rules are enforced by a Gradle plugin checked into buildSrc/ (the recommended location per this thread).
  • A class's effective audience is its own direct @InterfaceAudience annotation if present, otherwise the audience of its nearest annotated enclosing class. Default everywhere is @Private per the KIP. A  nested class can override an inherited @Public with an explicit @InterfaceAudience.Private. This matches Hadoop's model and aligns with how javadoc treats nested classes — they're documented under the outer's page, so requiring a separate annotation on every public nested class would be redundant noise. Inheritance also handles protected and package-private nested classes correctly: they show up in javadoc only as part of the outer's docs, so the checker only enforces direct @Public for the MISSING_JAVADOC check, while cascade reference checks accept inherited @Public for any nested type.

Guardrails for Plugin developers

Two plugins are published as part of Apache Kafka releases. Both operate on compiled bytecode rather than source-level imports, so they catch Java, Scala, Kotlin, and fully-qualified usages uniformly. The previous source-regex approach is replaced.

The scanner walks every .class reachable from the configured roots via an ASM ClassVisitor and records every reference into org.apache.kafka.** — including superclasses/interfaces, field types, method signatures, method-body instructions (NEW, INVOKE*, GETFIELD, CHECKCAST, ANEWARRAY, INVOKEDYNAMIC, class literals, generic signatures, type annotations). Each reference is checked against the published API surface; references to classes not annotated @InterfaceAudience.Public (and not covered by @SuppressKafkaInternalApiUsage) fail the build.

Known limitation: static final compile-time constants (primitives and String) are inlined by javac per JLS §13.1, so the referenced class is absent from the bytecode. This is an intrinsic property of any bytecode-based scanner.

Gradle plugin

Purpose: Prevents external Gradle-based projects from accidentally using internal Kafka APIs.

How to use:


plugins {
    id 'org.apache.kafka.internal-api-checker' version 'X.X.X'
}

kafkaInternalApiChecker {
    // Enable/disable the checker (default: true)
    enabled = true

    // Fail build on violations (default: true)
    failOnViolation = true

    // Compiled-class directories or jars to scan.
    // Default: build/classes (covers java/, scala/, kotlin/ output subdirs)
    classDirs = files("$buildDir/classes")

    // Report file location
    reportFile = file("$buildDir/reports/kafka-internal-api-usage.txt")
}

The task is named kafkaInternalApiChecker, depends on the project's classes task (so compilation runs first), and is wired into check.

Maven plugin


<plugin>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-internal-api-checker-maven-plugin</artifactId>
    <version>X.X.X</version>
    <configuration>
        <enabled>true</enabled>
        <failOnViolation>true</failOnViolation>

        <!-- Defaults to ${project.build.outputDirectory} and
             ${project.build.testOutputDirectory} when omitted. -->
        <classesDirectories>
            <classesDirectory>${project.build.outputDirectory}</classesDirectory>
            <classesDirectory>${project.build.testOutputDirectory}</classesDirectory>
        </classesDirectories>

        <reportFile>${project.build.directory}/reports/kafka-internal-api-usage.txt</reportFile>
    </configuration>
    <executions>
        <execution>
            <phase>verify</phase>
            <goals><goal>verify</goal></goals>
        </execution>
    </executions>
</plugin>

Suppression example


import org.apache.kafka.common.annotation.SuppressKafkaInternalApiUsage;

@SuppressKafkaInternalApiUsage("ports legacy adapter; tracked in JIRA-1234")
public class LegacyAdapter {
    // references to internal Kafka classes here are skipped by the checker
    // and surfaced in the report with the reason above
}

Report format

The checker emits three artefacts: a text report, a JSON report, and a coloured console summary. All three split entries into two sections:


Apache Kafka Public API Violation Report
========================================
Generated: 2026-05-13T11:03:05
Total violations: 0
Total suppressions: 7

## Suppressions (7 entries)
References skipped due to @SuppressKafkaInternalApiUsage on the consumer.
Each line shows the reason supplied to the annotation; review periodically.

- Suppressed reference to internal Kafka class org.apache.kafka.clients.producer.ProducerRecord
  from com.mycompany.app.App#main (line 48) — reason: ports legacy adapter; tracked in JIRA-1234
...

The build fails only on entries in the violations section; suppressions are informational but persisted so reviewers can audit them in PR review.


Compatibility, Deprecation, and Migration Plan

  • The annotations are only consulted at build time; no runtime behaviour changes.
  • During the migration window, classes that lack @InterfaceAudience.Public are treated as Private. The migration script seeds every Javadoc'd class with @InterfaceAudience.Public so the on-trunk surface is unchanged at flip-over.
  • External plugin authors with existing internal-API usages can adopt the checker incrementally by adding @SuppressKafkaInternalApiUsage("…") at the necessary sites; each suppression is visible in the report for later cleanup.
  • Plugin configuration migrated from source-directory inputs to class-directory inputs (Gradle: sourceDirs → classDirs; Maven: <sourceDirectories> → <classesDirectories>). Defaults are sensible (build/classes / ${project.build.outputDirectory}), so projects that relied on defaults need no change.

Test Plan

  • Unit tests — BytecodeApiUsageScannerTest covers empty input, references to public-annotated classes (skipped), references to internal classes (flagged), JDK refs ignored, jar inputs, field-type references, class-level suppression with reason captured, method-level suppression (only that method), and reason-less suppression.
  • End-to-end gradle test — sample consumer app (my-simple-app) referencing KafkaProducer, ProducerRecord, Callback, RecordMetadata, StringSerializer. Run produces 7 violations on internal types; KafkaProducer (the only @InterfaceAudience.Public class) is correctly skipped. Adding class-level @SuppressKafkaInternalApiUsage flips to BUILD SUCCESSFUL with 7 suppressions logged.
  • End-to-end maven test — same consumer compiled with the Maven plugin, identical results.
  • Existing Kafka tests continue to pass.

Rejected Alternatives

@InterfaceAudience.LimitedPrivate({"Connect", "Streams"})

Hadoop offers a third audience for "public to a specific subsystem, private to everyone else." We considered borrowing it but rejected it because:

  1. Externally we cannot verify the caller's claimed audience without intrusive consumer-side build config.
  2. InterfaceStability already encodes the contract-narrowing signal.
  3. Two values (Public, Private) keep the mental model simple. Subsystem-specific access can be documented in Javadoc or enforced via package boundaries.

Baseline file for known violations

A baseline file (one violation key per line, generated with a --update-baseline flag) would let projects with existing legitimate exposures adopt the checker without source changes. Rejected for the initial version because annotations are reviewable inline (PRs surface them) and carry a reason. A baseline file may be revisited if onboarding friction proves significant.

Build-config allowlist

kafkaInternalApiChecker { ignore = [...] } would centralise exceptions in the build script. Rejected because it drifts silently as code moves, carries no per-violation reason, and duplicates what the annotation already provides.

Source-level .java regex scan

The first prototype scanned .java source files for import org.apache.kafka.* lines. Replaced by the ASM bytecode scan because the source approach (a) misses Scala/Kotlin consumers entirely, (b) misses fully-qualified usages with no import, (c) is brittle to formatting. The bytecode walk catches all JVM-language consumers uniformly.


Amendments (implementation deltas, post-acceptance)

During implementation (apache/kafka#21337) four points in the KIP diverged from the approved text. Each is intentional, has reviewer agreement, and is captured here so the KIP reflects what shipped.

1. Report format: JSON output dropped, text + console only

The “Report Format” section originally listed three outputs — plain text, JSON, and colored console. Only text and console are implemented; no JSON.

Rationale. No downstream consumer surfaced during implementation that needed a stable JSON wire format. Maintaining a third schema (and a test-and-evolution contract around it) for no current consumer is cost without benefit. The text report is already line-oriented enough to grep, and the console output is what humans actually read. If a future consumer needs a machine-readable format, it can be added under its own KIP with a versioned schema.

2. Default scan scope is main classes only, not main+test

The “Configuration” section originally defaulted Maven’s <classesDirectories> to both ${project.build.outputDirectory} and ${project.build.testOutputDirectory}, and Gradle’s classDirs to files("$buildDir/classes") (both source sets). The implementation defaults to main classes only; users who want to scan test code opt in explicitly.

Rationale. Test code legitimately reaches into internal Kafka utilities (e.g. embedded brokers, mock clients, internal-only helpers under org.apache.kafka.test.*). Scanning test bytecode by default would produce noise that isn’t a real consumer-side concern — plugin and connector users don’t ship their tests. The checker is also not the right gate for “did you import something internal in a test”; that’s between the developer and their reviewer.

The Gradle plugin’s default is now sourceSets.main.output.classesDirs and the Maven mojo’s default is ${project.build.outputDirectory} only. Override with classesDirectories (Maven) or classDirs (Gradle) to include test outputs.

3. Report header: no Generated: <timestamp> line

The “Report Format” example originally showed a leading Generated: <ISO timestamp> line. The implementation omits it.

Rationale. Byte-reproducible report output. Build systems hash the report contents to decide whether downstream tasks need to re-run; a timestamp on the first line invalidates that hash on every build for no information gain. The report’s content already reflects the inputs that produced it, and the build timestamp is available from the surrounding build log if anyone needs it.

4. Implementation location: api-checker/ composite-included build, not buildSrc/

The KIP described placing the checker plugins under buildSrc/. They live instead in a dedicated api-checker/ directory that’s composite-included from the root settings.gradle via pluginManagement { includeBuild 'api-checker' }. Three subprojects (api-checker/core, api-checker/gradle-plugins, api-checker/maven-plugin) publish their own Maven coordinates.

Rationale. A buildSrc/-resident plugin shares its classpath with Kafka’s main build. The Maven mojo brings maven-plugin-api, maven-core, maven-artifact, and maven-plugin-annotations as runtime dependencies — none of which any other part of Kafka needs. Hosting them under buildSrc/ would put those jars on every developer’s main-build classpath and inflate the producer artifacts. The composite-included layout also lets each published jar carry only the classes its consumers need: the Gradle plugin jar doesn’t ship the Maven mojo classes, the Maven plugin jar doesn’t ship the Gradle task classes, and both share a thin kafka-api-checker-core jar that carries only the ASM-based scanner.


  • No labels