Status

Current state: Under Discussion

Discussion thread: thread-1 and thread-2

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Apache Kafka currently specifies that any class which publishes Javadoc is Public - Kafka Improvement Proposals#Whatisconsidereda%22majorchange%22thatneedsaKIP. To designate an interface as a public API, implementers should define a Gradle rule following the pattern established in build.gradle.

However,  there is a risk that builders may inadvertently import / expose internal classes . For example, see KIP-1247: Make Bytes utils class part of the public API

This KIP proposes to introduce a mechanism that 

• Explicitly declares Public APIs.

• Restricts the unintentional exposure of internal interfaces.

• Enables automated detection of internal API usage for plugin developers.

Public Interfaces

We propose to define a new interface which will be used to annotate public classes.

• org.apache.kafka.common.annotation.PublicAPI 

Any class without this annotation will be considered internal.
Note that Apache Kafka already has annotations for denoting stability of an API , which will continue to be used for the same purpose. Eg:

@PublicAPI
@InterfaceStability.Evolving
public class AlterClientQuotasResult {

// a public API whose interface might change in the future
}

Proposed Changes

The current definition mentions that any class with a JavaDoc comment is considered to be part of public API. We will run a script to identify all the classes which currently has javadoc definitions and then annotate those classes using the new interface . 

Restrictions in Apache Kafka code

We will also add guard rails to restrict public API parameters and return values to publicly accessible classes only.

We will use/implement a Gradle plugin similar to  https://github.com/revelc/apilyzer-maven-plugin to add these restrictions . Eg: https://code.revelc.net/apilyzer-maven-plugin/#annotation-example . This plugin implementation will also be checked into Apache Kafka code in the buildSrc  directory (as suggested by this thread).

The docsJar step will fail if the Javadoc jar contains any un-annotated classes. Additionally, any class marked with @PublicApi must be included in the docsJar or an error will be reported.

Guardrails for Plugin developers

The following plugins will be published to Maven as part of Apache Kafka release.

GradleKafkaPluginInternalApiChecker (Publishable Gradle Plugin)

Purpose: Prevents external Gradle-based projects from accidentally using internal Kafka APIs.

Behavior:

• Can be added to any Gradle build by external plugin developers
• Scans all imports from org.apache.kafka.** packages
• Validates that imported classes have the @PublicApi annotation
• Fails the build if any imported Kafka class lacks the @PublicApi annotation

How to use:

plugins {
    id 'org.apache.kafka.internal-api-checker' version 'X.X.X'
}

kafkaInternalApiChecker {
    // Enable/disable the checker (default: true)
    enabled = true

    // Fail build on violations (default: true)
    failOnViolation = true

    // Kafka version to validate against (default: auto-detected from dependencies)
    kafkaVersion = '3.6.0'

    // Source directories to scan (default: src/main/java)
    sourceDirs = [file('src/main/java'), file('src/test/java')]

    // Report file location
    reportFile = file("${buildDir}/reports/kafka-internal-api-usage.txt")
}

MavenKafkaPluginInternalApiChecker (Publishable Maven Plugin)

A similar plugin for Maven also will be published which can be configured as follows -

<plugin>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-internal-api-checker-maven-plugin</artifactId>
    <version>X.X.X</version>
    <configuration>
        <!-- Enable/disable the checker (default: true) -->
        <enabled>true</enabled>

        <!-- Fail build on violations (default: true) -->
        <failOnViolation>true</failOnViolation>

        <!-- Kafka version to validate against -->
        <kafkaVersion>3.6.0</kafkaVersion>

        <!-- Source directories to scan -->
        <sourceDirectories>
            <sourceDirectory>src/main/java</sourceDirectory>
            <sourceDirectory>src/test/java</sourceDirectory>
        </sourceDirectories>

        <!-- Report file location -->
        <reportFile>${project.build.directory}/reports/kafka-internal-api-usage.txt</reportFile>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>check</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Compatibility, Deprecation, and Migration Plan

There should be no impact to existing features as the annotations are only used during build.

Test Plan

• All existing tests should pass
• Build should fail if any new code is using internal API classes

Rejected Alternatives