Status

Current state: Accepted

Discussion thread: here

JIRA: here

Motivation

This KIP builds on top of KIP-975 and aims to introduce a JVM-based Docker Official Image (DOI) for Apache Kafka that will visible under Docker Official Images.

Once implemented for Apache Kafka, for each release, there will be one more JVM-based Docker image available to users.

We already have an OSS sponsored image, which was introduced via KIP-975 (apache/kafka) which comes under The Apache Software Foundation in Docker Hub.

The new Docker Image is the Docker Official Image (DOI), which will be built and maintained by Docker Community.

For a release version like 3.8.0 we will have two JVM based docker images:-

• apache/kafka:3.8.0 (OSS sponsored image)
• kafka:3.8.0 (Docker Official image)

Several other Apache projects, like Flink, Spark, Solr, have already released Docker Official Images, with download figures ranging from 50 million to over 1 billion. These numbers highlight the significant demand among users. Also the Docker Official Images are always the top 1 search result, irrespective of the number of downloads. This visibility guarantees that over time, a larger number of users will opt for Docker images provided by the Apache Kafka community. The support from these Apache projects indicates that we can achieve similar success with Apache Kafka.

The Dockerfile for Apache Kafka will continue to be managed by the Apache Kafka community and undergo review by Docker Community. Once approved by Docker community, the Docker Official Image will be built and published by the Docker Community.

We will recommend using the official images on Docker Hub, as they are reviewed by Docker. The images on apache/kafka will be provided in case of delays in the review process by Docker.

Docker Official Images are considered more secure than OSS sponsored images due to their strict and transparent build processes.

They are maintained through a collaboration between upstream maintainers, community volunteers, and Docker engineers, ensuring adherence to best practices.

The build process is open and transparent, with all changes undergoing public scrutiny through pull requests, and build logs available for inspection.

Each image also comes with a complete Software Bill of Materials (SBOM) and and detailed build provenance as signed attestations, providing users with detailed insight into the image's origins and security.

Regular security audits, updates, and the integration of signature validation in the image pull and build processes further enhance security. Source: https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/

Its these benefits that the Docker Official Images enjoy, thus making them more a more secure option for users to deploy Apache Kafka. Furthermore, the Docker Official image lowers the barrier to entry for new Apache Kafka users who might not be familiar with Apache or the concept of OSS images. We aim to simplify the process for Kafka beginners on Docker Hub, to discover, find and use the Kafka image.

* Unsupported yet.

** To be created.

Public Interfaces

• There will be a new additional artifact i.e. Official Kafka docker image for every Apache Kafka release.

• Add public documentation in Official Docker Image Repo.

Proposed Changes

• There will be a docker official image as an additional artifact for every Apache Kafka release. The docker files for the same will be maintained by the Apache Kafka community, but the images will be maintained by the Docker community.

• This docker official image, like the existing OSS sponsored images, will consist of JVM based Kafka and will have support for Linux based AMD and ARM architectures. 

• Add steps to the post-release process to include release of docker official image.

Compatibility, Deprecation, and Migration Plan

• No impact

Making the Dockerfile Compliant

• We will need to make sure to carry out the necessary modifications to ensure our Dockerfile aligns with the guidelines and criteria established for Official Docker Images, and ensure that it adheres to Docker's best practices. We will take steps to ensure our image aligns with the below pointers.

  • The New Image checklist - NEW-IMAGE-CHECKLIST.md

  • Best practices for Dockerfile instructions - Best practices for Dockerfile instructions

  • General best practices for writing Dockerfiles - General best practices for writing Dockerfiles

  • Review Guidelines: https://github.com/docker-library/official-images/blob/master/README.md#review-guidelines

• This step is necessary because the Dockerfiles will be subject to review by the Dockerhub team, and will require back-and-forth communication with them.

Proposed Workflow for Release Process

1. Initial Setup:

   • Establish a new directory, docker/docker_official_images, within trunk to keep Docker Official Image assets.

   • Ensure the Dockerfile is compliant as per Docker Official Images guidelines.

The following steps are to be followed once there is a new major_version.minor_version.patch_version that has already been voted on and released. As part of the release process, RM can create a JIRA for this task. These actions can be carried out by the RM or any contributor post the release process.

1. Latest major_version.minor_version.patch_version release:

   • Once a major_version.minor_version.patch_version is officially released and the binary URL for the release is available:

   • For all subsequent steps - use the official binary URL for the latest Apache Kafka major_version.minor_version.patch_version release, having the Scala version 2.13, since we are only supporting scala version 2.13 for docker images as per KIP-975. For instance, use https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz for the 3.7.0 release.

2. Docker Image Preparation via GitHub Actions:

   • Implement a GitHub Actions workflow to:

     • Generate a version-specific folder, following the major_version.minor_version.patch_version naming scheme, under the docker/docker_official_images directory, for housing static Dockerfile and scripts. For example 3.7.0.

     • This folder will contain all static files necessary for creating the docker image.

     • The github actions workflow generates this folder which can be downloaded and verified.

   • The RM puts this folder under the docker/docker_official_images directory in their fork of the kafka repo, and merges this change into the official kafka repo by raising a PR.

3. Docker Build and Release Testing:

   • Run a GitHub Actions workflow to perform Docker build tests to ensure image integrity. The dockerfiles for the same will be used from the specific folder in the docker/docker_official_images directory.

4. Docker Hub Submission:

   • Remove unsupported version folders from the docker_official_images directory to maintain repository cleanliness.

   • Run a script from trunk to automate the creation of a file which contains all the necessary information for submission to the Docker Hub official images library.

   • The RM will manually raise the final PR to Docker Hub’s official images repository using the contents of the generated file. Following PR approval, it is now Docker’s responsibility to build and publish the Docker Official Images. For example kafka:3.8.0.

5. Post Release Process - if Dockerhub folks suggest changes to the Dockerfiles:

   • Important:- Make sure the change suggested by Dockerhub is only specific to build and not a functionality change. In case it's a functionality change request then it would require a new KIP and the below mentioned steps will not be applicable.

   • Make the same changes in the Dockerfile present in docker/jvm/Dockerfile, so that the changes are propagated to newer RC releases and new major, minor and patch releases. This ensures that the source of truth is kept updated. Additionally, update the Dockerfile in the docker official image for that specific release. Commit these changes.

   • The RM will cherry pick these changes into the branch of that particular release.

   • Run the build and test workflow to test the images with the latest changes.

   • If the image builds successfully, and passes all tests, follow step 4 again to raise a new PR in the docker official images repo.

Directory Structure

A new directory named docker_official_images will be added to the /docker repository. This directory will contain all the Official Docker Images related code.

kafka/
    - docker/
        - docker_official_images/ #Contains resources needed to create the official docker images.
            - 3.8.1/              #Contains resources needed to create the official docker images for the 3.8.1 version.
              - jvm/              #Contains Dockerfile for the JVM-based Apache Kafka Official Docker image.
                - Dockerfile + necessary scripts
            - 3.9.0/
              - jvm/
                - Dockerfile + necessary scripts
        - jvm/
            - Dockerfile          #Dockerfile for the JVM-based Apache Kafka Docker image.
        - resources/              #Contains resources needed to create the Docker image.
        - test/                   #Contains sanity tests for the Docker image.
        - docker_build_test.py    #Python script for building and testing the Docker image.
        - docker_release.py       #Python script for building the Docker image and pushing it to Docker Hub.
        - docker_build_latest_artifact.py #Python script for building the artifact for the current RC release.
        - generate-stackbrew-library-docker.sh #Shell script to automate PR template for Docker Official Images.

Dockerhub Repo PR

RM will run a script provided in the docker folder to generate a file which can be used to create a PR on https://github.com/docker-library/official-images/tree/master
The final file will look like the following:-

Maintainers: The Apache Kafka Project <dev@kafka.apache.org> (@ApacheKafka)
GitRepo: https://github.com/apache/kafka.git

Tags: 3.8.1
Architectures: amd64,arm64v8
GitCommit: 9o7q27wmmu5kzh0erwr2rmvr5h7pdjh8a56a77u8
Directory: ./docker/docker_official_images/3.8.1/jvm

Tags: 3.9.0, latest
Architectures: amd64,arm64v8
GitCommit: 2w6tit74aog1o6w2m0jvfk7xtyd6qdep9opp00fd
Directory: ./docker/docker_official_images/3.9.0/jvm

Test Plan

Existing tests for docker image should pass.

Rejected Alternatives

Creating a separate repository named apache-docker within the Apache GitHub organization. This strategy mirrors the practices of other Apache projects, such as Spark and Flink, where Docker-related files and scripts reside in a distinct repository. However, this move would involve transferring existing Docker resources to the new repository too, without providing any added advantage. This approach does not provide any benefit in the requirement for creating specific folders with hardcoded values for each Docker image release.

While planning how to release Docker Official Images (DOI) as part of Kafka's release process, another approach involved integrating it as part of the release process itself. However, this approach made things much harder for the RMs. There were many more manual steps needed to be done by the RM which made the release process more complicated. By handling the release of Docker Official Images (DOIs) as a post-release process, we kept the release process unchanged and simplified the DOI release. This way, releasing DOIs can be easily managed by anyone after the main release is completed.