...

Proposed Changes

This KIP proposes to optionally include fenced brokers in the response to DescribeClusterRequest . A new boolean field,  IncludeFencedBrokers , will be added to DescribeClusterRequest. By default, fenced brokers will not with version 2 or later.  This means all brokers, fenced and unfenced will be included in the list of broker nodes returnedin the response. Also a new boolean field,   "Fenced,  " will be added to each broker's information in DescribeClusterResponse.

Currently, if EndpointType in the DescribeClusterRequest is 2 (controllers), a list of registered controller nodes are returned using the same DescribeClusterBroker class.  The DescribeClusterBroker class will be updated with the new field, "fenced". Therefore when returning controller nodes in the response, this field would just be set to false as the default value since this field is not relevant for controller nodes. 

Public Interfaces

DescribeClusterRequest: v2

{
  "apiKey": 60,
  "type": "request",
  "listeners": ["zkBroker", "broker", "controller"],
  "name": "DescribeClusterRequest",
  //
  // Version 1 adds EndpointType for KIP-919 support.
  // Version 2 adds IncludeFencedBrokers for an additional field in the response and the request is unchanged (KIP-1073 support).
  //
  "validVersions": "0-2",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "IncludeClusterAuthorizedOperations", "type": "bool", "versions": "0+",
      "about": "Whether to include cluster authorized operations." },
    { "name": "EndpointType", "type": "int8", "versions": "1+", "default": "1",
      "about": "The endpoint type to describe. 1=brokers, 2=controllers." },
     // NEW FIELD
    { "name": "IncludeFencedBrokers", "type": "bool", "versions": "2+",
      "about": "Whether to include fenced brokers." }
  ]
}

DescribeClusterResponse: v2

{
  "apiKey": 60,
  "type": "response",
  "name": "DescribeClusterResponse",
  //
  // Version 1 adds the EndpointType field, and makes MISMATCHED_ENDPOINT_TYPE and
  // UNSUPPORTED_ENDPOINT_TYPE valid top-level response error codes.
  // Version 2 adds Fenced field to Brokers for KIP-1073 support.
  //
  "validVersions": "0-2",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "ErrorCode", "type": "int16", "versions": "0+",
      "about": "The top-level error code, or 0 if there was no error" },
    { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "default": "null",
      "about": "The top-level error message, or null if there was no error." },
    { "name": "EndpointType", "type": "int8", "versions": "1+", "default": "1",
      "about": "The endpoint type that was described. 1=brokers, 2=controllers." },
    { "name": "ClusterId", "type": "string", "versions": "0+",
      "about": "The cluster ID that responding broker belongs to." },
    { "name": "ControllerId", "type": "int32", "versions": "0+", "default": "-1", "entityType": "brokerId",
      "about": "The ID of the controller broker." },
    { "name": "Brokers", "type": "[]DescribeClusterBroker", "versions": "0+",
      "about": "Each broker in the response.", "fields": [
      { "name": "BrokerId", "type": "int32", "versions": "0+", "mapKey": true, "entityType": "brokerId",
        "about": "The broker ID." },
      { "name": "Host", "type": "string", "versions": "0+",
        "about": "The broker hostname." },
      { "name": "Port", "type": "int32", "versions": "0+",
        "about": "The broker port." },
      { "name": "Rack", "type": "string", "versions": "0+", "nullableVersions": "0+", "default": "null",
        "about": "The rack of the broker, or null if it has not been assigned to a rack." },
      // NEW FIELD
      { "name": "Fenced", "type": "bool", "versions": "2+", 
        "about": "Whether the broker is fenced." }
    ]},
    { "name": "ClusterAuthorizedOperations", "type": "int32", "versions": "0+", "default": "-2147483648",
      "about": "32-bit bitfield to represent authorized operations for this cluster." }
  ]
}

AdminClient

...

A new field, includeFencedBrokers, will be added to DescribeClusterOptions. By default, this option is set to false. When includeFencedBrokers is set to true, any fenced brokers will be included in the DescribeClusterResult returned by Admin clients. 

DescribeClusterOptions

...

...

There will not be any significant change for Admin client when describing a cluster. However the Node class used for reading the DescribeClusterResponse data will be updated with a new field, "fenced". If the response from the broker did not include fenced brokers and the new "fenced" field in broker description, the field in the Node class will be set to false as the default. 

kafka-cluster.sh

The console tool used for describing cluster will be updated with a new command to list brokers and a parameter to include fenced brokers. nodes. When it is used with --bootstrap-server, the output will include STATE column to describe whether a broker is fenced. When it is used with --bootstrap-controller, the output will not include the STATE column as this is not relevant for controller nodes. 

Example:

./bin/kafka-cluster.sh --bootstrap-server localhost:9092 list-brokersnodes
ID         HOST      PORT       STATE      RACK       
0          broker-0  9092       unfenced              
1          broker-1  9092       unfenced
2          broker-2  9092       fenced             


./bin/kafka-cluster.sh --bootstrap-servercontroller localhost:90929093 list-brokers --include-fenced-brokersnodes
ID         HOST      		PORT       STATE      RACK       
0          broker controller-03  9192	9093       unfenced              
1          broker controller-14  9092     	9093  unfenced    
2          broker controller-25  9092 	9093      fenced   // (Broker 2 has shutdown but still registered)

Compatibility, Deprecation, and Migration Plan

• Existing Admin clients using older versions will not request fenced brokers in DescribeCluster requests and the default is to return unfenced brokers only. This keeps older clients compatible with newer brokersonly receive the unfenced brokers since they will be using the older protocol version.
• Newer Admin clients connecting to older brokers will not receive the fenced brokers will use the older protocol version and hence will not request fenced brokersin the response. The new field in Node class, "fenced" will still be populated when reading the response data and will be set to false for the brokers returned.

Rejected Alternatives

Returning inactive observer nodes in DescribeMetadataQuorumResponse was considered. However, observers information is only stored in the quorum leader's memory state, therefore this information is lost when there is a leader change. 

...