Status

Current state: "Draft"

Discussion thread: here

JIRA: KAFKA-5639

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The DescribeGroups protocol v1 currently returns this information for each consumer group:

• error_code
• group_id
• state
• protocol_type
• protocol
• members (only a member summary is returned per member that misses some useful member info too)

Additional info exist or can be derived in a GroupMetadata object on the server side, some of which could be useful if exposed via the DescribeGroups protocol. Here are some examples:

• generationId
• leaderId
• supportedProtocols per member
• subscribed topics per member

Enhancing the protocol with some of this additional info means improving the existing tools that make use of it. For example, using this additional info, the consumer group command's --describe output will provide more information about each consumer group to help with understanding the group behavior, its monitoring, troubleshooting, etc. 

Public Interfaces

This is the latest version (v1) of DescribeGroups protocol:

DescribeGroups Request (Version: 1) => [group_ids] 
  group_ids => STRING
 
DescribeGroups Response (Version: 1) => throttle_time_ms [groups] 
  throttle_time_ms => INT32
  groups => error_code group_id state protocol_type protocol [members] 
    error_code => INT16
    group_id => STRING
    state => STRING
    protocol_type => STRING
    protocol => STRING
    members => member_id client_id client_host member_metadata member_assignment 
      member_id => STRING
      client_id => STRING
      client_host => STRING
      member_metadata => BYTES
      member_assignment => BYTES

The suggestion is to enhance the protocol to version 2 and add some of these currently missing information (* indicates suggested fields to be added to the protocol):

DescribeGroups Request (Version: 2) => [group_ids] 
  group_ids => STRING


DescribeGroups Response (Version: 2) => throttle_time_ms [groups] 
  throttle_time_ms => INT32
  groups => error_code group_id state protocol_type protocol generation_id leader_id new_member_added received_consumer_offset_commits received_transactional_offset_commits [members] 
    error_code => INT16
    group_id => STRING
    state => STRING
    protocol_type => STRING
    protocol => STRING
  * generation_id => INT32
  * leader_id => STRING
    members => member_id client_id client_host member_metadata member_assignment member_subscription [member_protocols]
      member_id => STRING
      client_id => STRING
      client_host => STRING
      member_metadata => BYTES
      member_assignment => BYTES
  *   member_subscription => BYTE
  *   member_protocols => STRING

Proposed Changes

The proposal is to

1. Create version 2 of DescribeGroups protocol to include additional information about the consumer group and also each group member for the API clients.
2. Update the consumer group command to provide the added information in the command output (where applicable). Note that this KIP will improve upon KIP-175 where group state and detailed member information will also be returned by the consumer group command using additional command line options. Therefore, the output of some of the sub-options introduced in KIP-175 will improve to include the newly available information by this KIP.

Compatibility, Deprecation, and Migration Plan

Since this KIP follows KIP-175, the output for sub-options --members and --state that are introduced in that KIP will be modified to also include the newly added fields. Therefore, clients who rely on the output for those sub-options may need to be adjusted. It also helps the consumer group --describe command to fill some of the newly added columns introduced by KIP-175 (i.e. ASSIGNMENT-STRATEGY for --members options, and SUBSCRIPTION for --members --verbose option).

Rejected Alternatives

There are other information available for a consumer group that could be included in the KIP. For example:

• newMemberAdded

• receivedConsumerOffsetCommits

• receivedTransactionalOffsetCommits

Instead of adding this information to the protocol (and having to explain to users what it means and also having to continue exposing it in the future even if implementation details change) we could report them as log messages (when necessary) on the broker side.