Status

Current state: "Under Discussion"

Discussion thread: [DISCUSS] KIP-1149 Helm Chart for Apache Kafka

JIRA: KAFKA-6416

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Deploying an Apache Kafka cluster to Kubernetes is a complex process and a Helm chart can greatly simplify this, making it possible to have a cluster up and running in a few minutes. There are third party charts available for example

These all use their own bespoke container images and often come with various other components to make more of a managed service offering.

This proposal if for a relatively straightforward Helm chart that will use the Apache published Docker container image. It will enable a production ready Kafka cluster to be set up with a few easy steps.

It will allow the full set of properties to be configured by setting helm values, so that these can be managed without having to modify environment values in the container.

The Helm chart will only support deploying Kafka 4.1.0 and later. I did have it working with 3.9.0, but changes to logging configuration would make this complex to support moving forward.

Public Interfaces

This change will not add or change any interfaces the application has. The Helm chart is separate to the Java application.

Proposed Changes

A draft PR with proposed changes is at PR #19249

The new Helm chart is contained in the /chart folder.

The properties files will be templated by the Helm chart and stored in Kubernetes as ConfigMaps. These will be mounted into the running containers as read-only files. A new shell script file , k8s_launch , will be added to the Dockerfile as the start up script to run when deployed with the Helm chart. This script will handle copying the mounted properties files to the correct locations and making them writeable. It will also handle the formatting of the storage.

This additional script file should be the only change needed to the existing codebase. Compatibility, Deprecation, and Migration Plan

The change should have no impact on existing users. This will be a additional script file in the docker container, but if not deployed with the new Helm chart, this would not be used.

The CD pipeline would need to be changed to build and publish the Helm chart with Kafka. An example of an Apache project that publishes it's own Helm chart is Apache Airflow. The chart repository index is in the file airflow.apache.org/index.yaml and the charts are downloaded from downloads.apache.org like any other binary.

Test Plan

Deploy Kubernetes to a few different Kubernetes distributions (and Cloud services) and run any existing system validation tests.

CI could be changed to run the Helm lint command and the Helm template command is useful to validate the templating works.

Rejected Alternatives

Use of existing third party deploying solutions such as Confluent or Bitnami, as listed above.

  • No labels