...
Current state: "Under Discussion"
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Proposed Changes
This KIP proposes to optionally include fenced brokers in the response to DescribeClusterRequest . A new boolean field, includeFencedBrokers , will be added to DescribeClusterOptions. By default, fenced brokers will not with version 2 or later. This means all brokers, fenced and unfenced will be included in the list of broker nodes returnedin the response. Also a new boolean field, isFenced, "Fenced" will be added to each broker's information in DescribeClusterResponse.
Currently, if EndpointType in the DescribeClusterRequest is 2 (controllers), a list of registered controller nodes are returned as the DescribeClusterBroker in the result. The DescribeClusterBroker class will be updated with the new field, "fenced". Therefore when returning controller nodes in the response, this field would just be set to false as the default value since this field is not relevant for controller nodes.
Public Interfaces
DescribeClusterRequest: v2
| Code Block | ||
|---|---|---|
| ||
{
"apiKey": 60,
"type": "request",
"listeners": ["zkBroker", "broker", "controller"],
"name": "DescribeClusterRequest",
//
// Version 1 adds EndpointType for KIP-919 support.
// Version 2 adds IncludeFencedBrokers for an additional field in the response and the request is unchanged (KIP-1073 support).
//
"validVersions": "0-2",
"flexibleVersions": "0+",
"fields": [
{ "name": "IncludeClusterAuthorizedOperations", "type": "bool", "versions": "0+",
"about": "Whether to include cluster authorized operations." },
{ "name": "EndpointType", "type": "int8", "versions": "1+", "default": "1",
"about": "The endpoint type to describe. 1=brokers, 2=controllers." },
//NEW FIELD
{ "name": "IncludeFencedBrokers", "type": "bool", "versions": "2+",
"about": "Whether to include fenced brokers when listing brokers." }
]
} |
DescribeClusterResponse: v2
| Code Block | ||
|---|---|---|
| ||
{
"apiKey": 60,
"type": "response",
"name": "DescribeClusterResponse",
//
// Version 1 adds the EndpointType field, and makes MISMATCHED_ENDPOINT_TYPE and
// UNSUPPORTED_ENDPOINT_TYPE valid top-level response error codes.
// Version 2 adds IsFencedFenced field to Brokers for KIP-1073 support.
//
"validVersions": "0-2",
"flexibleVersions": "0+",
"fields": [
{ "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
"about": "The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The top-level error code, or 0 if there was no error" },
{ "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+", "default": "null",
"about": "The top-level error message, or null if there was no error." },
{ "name": "EndpointType", "type": "int8", "versions": "1+", "default": "1",
"about": "The endpoint type that was described. 1=brokers, 2=controllers." },
{ "name": "ClusterId", "type": "string", "versions": "0+",
"about": "The cluster ID that responding broker belongs to." },
{ "name": "ControllerId", "type": "int32", "versions": "0+", "default": "-1", "entityType": "brokerId",
"about": "The ID of the controller broker." },
{ "name": "Brokers", "type": "[]DescribeClusterBroker", "versions": "0+",
"about": "Each broker in the response.", "fields": [
{ "name": "BrokerId", "type": "int32", "versions": "0+", "mapKey": true, "entityType": "brokerId",
"about": "The broker ID." },
{ "name": "Host", "type": "string", "versions": "0+",
"about": "The broker hostname." },
{ "name": "Port", "type": "int32", "versions": "0+",
"about": "The broker port." },
{ "name": "Rack", "type": "string", "versions": "0+", "nullableVersions": "0+", "default": "null",
"about": "The rack of the broker, or null if it has not been assigned to a rack." },
// NEW FIELD
{ "name": "IsFencedFenced", "type": "bool", "versions": "2+",
"about": "Whether the broker is fenced." }
]},
{ "name": "ClusterAuthorizedOperations", "type": "int32", "versions": "0+", "default": "-2147483648",
"about": "32-bit bitfield to represent authorized operations for this cluster." }
]
} |
AdminClient
...
DescribeClusterOptions
...
| language | java |
|---|
...
There will not be any significant change for AdminClient when describing a cluster. However the Node class used for reading the DescribeClusterResponse data will be updated with a new field, "fenced". If the response from the broker did not include fenced brokers and the new "fenced" field in broker description, the field in the Node class will be set to false as the default.
kafka-cluster.sh
The console tool used for describing cluster will be updated with a new command to list brokers and a parameter to include fenced brokers. nodes. When it is used with --bootstrap-server, the output will include STATE column to describe whether a broker is fenced. When it is used with --bootstrap-controller, the output will not include the STATE column as this is not relevant for controller nodes. Also if there is no rack information for any of the nodes, the RACK column will be omitted from the output.
Example:
| Code Block | ||
|---|---|---|
| ||
./bin/kafka-cluster.sh --bootstrap-server localhost:9092 list-brokersnodes ID HOST PORT STATERACK RACKSTATE 0 broker-0 9092 rack-a unfenced 1 broker-1 9092 rack-b unfenced 2 broker-2 9092 rack-c fenced ./bin/kafka-cluster.sh --bootstrap-servercontroller localhost:90929093 list-brokers --include-fenced-brokersnodes ID HOST PORT PORT STATE RACK 0 broker controller-03 9192 9093 unfenced 1 broker controller-14 9092 9093 unfenced 2 broker controller-2 5 9092 9093 fenced → (Broker 2 has shutdown but still registered) |
Compatibility, Deprecation, and Migration Plan
- Existing Admin clients using older versions will only receive the unfenced brokers since they will be using the older protocol version.
- Newer Admin clients connecting to older brokers will not receive the fenced brokers in the response. The new field in Node class, "fenced" will still be populated when reading the response data and will be set to false for the brokers returned.
Rejected Alternatives
Returning inactive observer nodes in DescribeMetadataQuorumResponse was considered. However, observers information is only stored in the quorum leader's memory state, therefore this information is lost when there is a leader change.