Status

Current state: "Under Discussion"

Discussion thread: here 

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

When a KRaft broker node shuts down, it is "fenced", but still registered in the controller. To completely remove KRaft-based broker nodes, they must first be unregistered via the Kafka Admin API.

Removing a node without unregistering causes various issues, such as a newly created partition will still get assigned to the removed replicas, or preventing metadata version updates after an upgrade. If this happens, the cluster admin/operator needs to get the node id to unregister it. However, admin cannot list the removed node using any of the APIs.

Proposed Changes

This KIP proposes to include fenced brokers in the response to DescribeClusterRequest with version 2 or later.  This means all brokers, fenced and unfenced will be included in the list of broker nodes in the response. Also a new boolean field, "Fenced" will be added to each broker's information in DescribeClusterResponse.

Currently, if EndpointType in the DescribeClusterRequest is 2 (controllers), a list of registered controller nodes are returned as the DescribeClusterBroker in the result.  The DescribeClusterBroker class will be updated with the new field, "fenced". Therefore when returning controller nodes in the response, this field would just be set to false as the default value since this field is not relevant for controller nodes. 

Public Interfaces

DescribeClusterRequest: v2

{
  "apiKey": 60,
  "type": "request",
  "listeners": ["zkBroker", "broker", "controller"],
  "name": "DescribeClusterRequest",
  //
  // Version 1 adds EndpointType for KIP-919 support.
  // Version 2 adds an additional field in the response and the request is unchanged (KIP-1073).
  //
  "flexibleVersions": "0+",
  "fields": [
    { "name": "IncludeClusterAuthorizedOperations", "type": "bool", "versions": "0+",
      "about": "Whether to include cluster authorized operations." },
    { "name": "EndpointType", "type": "int8", "versions": "1+", "default": "1",
      "about": "The endpoint type to describe. 1=brokers, 2=controllers." }
  ]
}

DescribeClusterResponse: v2

{
  "apiKey": 60,
  "type": "response",
  "name": "DescribeClusterResponse",
  //
  // Version 1 adds the EndpointType field, and makes MISMATCHED_ENDPOINT_TYPE and
  // UNSUPPORTED_ENDPOINT_TYPE valid top-level response error codes.
  // Version 2 adds Fenced field to Brokers for KIP-1073 support.
  //
  "validVersions": "0-2",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "ErrorCode", "type": "int16", "versions": "0+",
      "about": "The top-level error code, or 0 if there was no error" },
    { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "default": "null",
      "about": "The top-level error message, or null if there was no error." },
    { "name": "EndpointType", "type": "int8", "versions": "1+", "default": "1",
      "about": "The endpoint type that was described. 1=brokers, 2=controllers." },
    { "name": "ClusterId", "type": "string", "versions": "0+",
      "about": "The cluster ID that responding broker belongs to." },
    { "name": "ControllerId", "type": "int32", "versions": "0+", "default": "-1", "entityType": "brokerId",
      "about": "The ID of the controller broker." },
    { "name": "Brokers", "type": "[]DescribeClusterBroker", "versions": "0+",
      "about": "Each broker in the response.", "fields": [
      { "name": "BrokerId", "type": "int32", "versions": "0+", "mapKey": true, "entityType": "brokerId",
        "about": "The broker ID." },
      { "name": "Host", "type": "string", "versions": "0+",
        "about": "The broker hostname." },
      { "name": "Port", "type": "int32", "versions": "0+",
        "about": "The broker port." },
      { "name": "Rack", "type": "string", "versions": "0+", "nullableVersions": "0+", "default": "null",
        "about": "The rack of the broker, or null if it has not been assigned to a rack." },
      // NEW FIELD
      { "name": "Fenced", "type": "bool", "versions": "2+", 
        "about": "Whether the broker is fenced." }
    ]},
    { "name": "ClusterAuthorizedOperations", "type": "int32", "versions": "0+", "default": "-2147483648",
      "about": "32-bit bitfield to represent authorized operations for this cluster." }
  ]
}

AdminClient

There will not be any significant change for AdminClient when describing a cluster. However the Node class used for reading the DescribeClusterResponse data will be updated with a new field, "fenced". If the response from the broker did not include fenced brokers and the new "fenced" field in broker description, the field in the Node class will be set to false as the default. 

kafka-cluster.sh

The console tool used for describing cluster will be updated with a new command to list nodes. When it is used with --bootstrap-server, the output will include STATE column to describe whether a broker is fenced. When it is used with --bootstrap-controller, the output will not include the STATE column as this is not relevant for controller nodes. Also if there is no rack information for any of the nodes, the RACK column will be omitted from the output. 

Example: 

./bin/kafka-cluster.sh --bootstrap-server localhost:9092 list-nodes
ID         HOST      PORT       RACK      STATE       
0          broker-0  9092       rack-a	  unfenced              
1          broker-1  9092       rack-b    unfenced
2          broker-2  9092       rack-c    fenced             

./bin/kafka-cluster.sh --bootstrap-controller localhost:9093 list-nodes
ID         HOST      	   PORT       
0          controller-3    9093             
1          controller-4    9093      
2          controller-5    9093

Compatibility, Deprecation, and Migration Plan

  • Existing Admin clients using older versions will only receive the unfenced brokers since they will be using the older protocol version.
  • Newer Admin clients connecting to older brokers will not receive the fenced brokers in the response. The new field in Node class, "fenced" will still be populated when reading the response data and will be set to false for the brokers returned.

Rejected Alternatives

Returning inactive observer nodes in DescribeMetadataQuorumResponse was considered. However, observers information is only stored in the quorum leader's memory state, therefore this information is lost when there is a leader change. 


  • No labels