Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-20897 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

A bad broker deployment can affect any partition it hosts. For services targeting 3-4 9’s SLO, the cost of a regression is dominated by how many partitions were exposed to the new build before it was rolled back.
Today an operator has no way to bound that set: a new broker build takes whatever partitions the placer happens to assign it.

This KIP introduces an optional broker attribute, `pod`, which groups brokers into named subsets, and a replica placer that can confine a chosen fraction of partitions to a single pod.

Deploying a new build to the canary pod then exposes a bounded, known set of partitions, so the blast radius of a regression is decided in advance rather than discovered afterwards.

This cannot be expressed with `broker.rack`. Rack-awareness means "spread replicas across these groups"; the requirement here is the inverse, "confine these partitions to this group". The two cannot be collapsed without adding a mode flag to an existing config and changing its meaning.

Future improvement include extend canary isolation from broker only to producer/consumer/broker canary isolation, thus blast radius of a bad producer/consumer service deployment can be isolated with-in canary pod 

Proposed Changes

Broker pod

Each broker may declare an optional pod name. The value is supplied at broker registration, stored as part of cluster metadata, and returned to clients alongside the broker's host, port and rack, so that a client can observe which pod a broker belongs to.

A broker that declares no pod belongs to every pod. This keeps existing clusters working unchanged and allows pods to be introduced one broker at a time.

It also means isolation is only complete once every broker declares a pod. While unlabelled brokers remain, they are eligible to host partitions belonging to any pod, including the canary pod. A partially labelled cluster should be treated as an intermediate migration state: operators should label all brokers before enabling canary placement.

Pod isolation

An isolation rule associates a pod with a set of partition ids. The placement logic evaluates the configured rules for each partition being created:

• A partition matching exactly one rule has its replicas placed only on brokers in that rule's pod;
• A partition matching no rule has its replicas placed across the brokers of all pods that carry no rule.

Within the brokers selected for a partition, replica and leader selection is unchanged, and existing rack-aware behaviour is preserved. Isolation constrains which brokers are eligible; it does not change how replicas are
distributed among them.

Placement rules are evaluated when partitions are created, both at topic creation and when partitions are added to an existing topic. They are not applied retroactively to partitions that already exist.

Canary isolation

Canary isolation is the single rule this KIP configures, through two controller configurations:

• `canary.pod.name` names the pod that canary partitions are placed on.
• `canary.partition.interval` designates every Nth partition of each topic as a canary partition, taking the last partition of each consecutive group of N. A value of 0 disables canary placement entirely, and placement behaves exactly as it does without this feature.

Taking the last partition of each group rather than the first means a topic with fewer partitions than the interval has no canary partition, so small topics are never drawn into the canary pod.

Because rules apply only at partition creation, existing topics are unaffected until their partitions are reassigned.

The partition reassignment tool therefore accepts the same canary pod name and interval, which must match the controller configuration. A plan generated without them would place canary partitions by ordinary placement and silently undo isolation on the topics it moves.

Example:

6 brokers with replication factor 3, three brokers in each of two pods, with the canary pod named `canary` and an interval of 10:

Two of twenty partitions, 10% of the topic, are exposed to a build deployed only to the canary pod's brokers.

Public Interfaces

New Configurations

New broker configs

New controller configs

Protocol Changes

A nullable Pod field is added as a tagged field to four messages. All four already declare flexible versions, so no message version is bumped and clients that do not recognize the tag skip it.

All are

Client API Change

org.apache.kafka.common.Node gains:

public String pod()

Command line

kafka-reassign-partitions.sh --generate gains:

so that plans generated offline match what the controller would produce.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

• PodReplicaPlacerTest: placement with zero, one, and multiple pods; pod-less brokers; empty clusters; partition addition to existing topics; and the multiple-rule conflict error.
• CanarySpecTest: interval-to-predicate conversion, including 0 and boundary values.
• BrokerRegistrationTest, MetadataCacheTest, BrokerLifecycleManagerTest: propagation of the field through registration, the metadata cache, and Node
• KafkaConfigTest: the new broker config
• ReassignPartitionsUnitTest: the new command-line options.

Rejected Alternatives

• Reuse broker.rack - Rack has spread semantics; pod has confine semantics. Expressing both through one attribute requires a mode flag that changes the meaning of an existing config.
• Version-bumped regular fields instead of tagged fields - All four messages already have flexible versions, and the field is null on essentially every broker in clusters that do not use pods — the sparse case tagged fields exist for. A version bump would also require clients to negotiate the new version to see the field, which is strictly more work for identical reach.
• Express the canary fraction as a percentage.** A percentage must be converted to a partition stride, and any value that is not a unit fraction converts lossily: 0.15 becomes an effective 16.7%, and any value above 0.5 selects every partition. An integer interval states the same intent exactly.

Future work

Canary isolation on the client side — a partitioner that routes canary traffic to canary partitions, and a consumer assignor that pins designated consumer instances to them

The work is deferred to a follow-up KIP so that the placement work can land on its own.