...
Field | Description |
---|---|
Leader | The node id for the kafka broker currently acting as leader for this partition. If no leader exists because we are in the middle of a leader election this id will be -1. |
Replicas | The set of alive nodes that currently acts as slaves for the leader for this partition. |
Isr | The set subset of the replicas that are "caught up" to the leader |
Broker | The node id, hostname, and port information for a kafka broker |
Possible Error Codes
* UnknownTopic (3),
* LeaderNotAvailable (5),
* InvalidTopic (17),
* TopicAuthorizationFailed (29)
Produce API
The produce API is used to send message sets to the server. For efficiency it allows sending message sets intended for many topic partitions in a single request.
...
Field | Description |
---|---|
TopicName | The name of the topic this response entry is for. |
Partition | The id of the partition this response is for. |
HighwaterMarkOffset | The offset at the end of the log for this partition. This can be used by the client to determine how many messages behind the end of the log they are. |
MessageSetSize | The size in bytes of the message set for this partition |
MessageSet | The message data fetched from this partition, in the format described above. |
Possible Error Codes
* OFFSET_OUT_OF_RANGE (1),
* UNKNOWN_TOPIC_OR_PARTITION (3),
* NOT_LEADER_FOR_PARTITION (6),
* REPLICA_NOT_AVAILABLE (9),
* UNKNOWN (-1)
Offset API (AKA ListOffset)
This API describes the valid offset range available for a set of topic-partitions. As with the produce and fetch APIs requests must be directed to the broker that is currently the leader for the partitions in question. This can be determined using the metadata API.
...
Code Block |
---|
OffsetResponse => [TopicName [PartitionOffsets]] PartitionOffsets => Partition ErrorCode [Offset] Partition => int32 ErrorCode => int16 Offset => int64 |
Possible Error Codes
* UNKNOWN_TOPIC_OR_PARTITION (3)
* NOT_LEADER_FOR_PARTITION (6)
* UNKNOWN (-1)
Offset Commit/Fetch API
These APIs allow for centralized management of offsets. Read more Offset Management. As per comments on KAFKA-993 these API calls are not fully functional in releases until Kafka 0.8.1.1. It will be available in the 0.8.2 release.
...
Code Block |
---|
GroupCoordinatorResponse => ErrorCode CoordinatorId CoordinatorHost CoordinatorPort ErrorCode => int16 CoordinatorId => int32 CoordinatorHost => string CoordinatorPort => int32 |
Offset Commit Request
Possible Error Codes
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_GROUP (16)
* GROUP_AUTHORIZATION_FAILED (30)
Offset Commit Request
Code Block |
---|
v0 (supported in 0 |
Code Block |
v0 (supported in 0.8.1 or later)
OffsetCommitRequest => ConsumerGroupId [TopicName [Partition Offset Metadata]]
ConsumerGroupId => string
TopicName => string
Partition => int32
Offset => int64
Metadata => string
v1 (supported in 0.8.2 or later)
OffsetCommitRequest => ConsumerGroupId ConsumerGroupGenerationId ConsumerId [TopicName [Partition Offset TimeStamp Metadata]]
ConsumerGroupId => string
ConsumerGroupGenerationId => int32
ConsumerId => string
TopicName => string
Partition => int32
Offset => int64
TimeStamp => int64
Metadata => string
v2 (supported in 0.8.3 or later)
OffsetCommitRequest => ConsumerGroup ConsumerGroupGenerationId ConsumerId RetentionTime [TopicName [Partition Offset Metadata]]
ConsumerGroupId => string
ConsumerGroupGenerationId => int32
ConsumerId => string
RetentionTime => int64
TopicName => string
Partition => int32
Offset => int64
Metadata => string |
...
Code Block |
---|
v0, v1 and v2: OffsetCommitResponse => [TopicName [Partition ErrorCode]]] TopicName => string Partition => int32 ErrorCode => int16 |
Offset Fetch Request
Possible Error Codes
* OFFSET_METADATA_TOO_LARGE (12)
* GROUP_LOAD_IN_PROGRESS (14)
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_GROUP (16)
* ILLEGAL_GENERATION (22)
* UNKNOWN_MEMBER_ID (25)
* REBALANCE_IN_PROGRESS (27)
* INVALID_COMMIT_OFFSET_SIZE (28)
* TOPIC_AUTHORIZATION_FAILED (29)
* GROUP_AUTHORIZATION_FAILED (30)
Offset Fetch Request
Per the comment on
Jira | |||
---|---|---|---|
|
Jira | ||||
---|---|---|---|---|
|
...
Note that if there is no offset associated with a topic-partition under that consumer group the broker does not set an error code (since it is not really an error), but returns empty metadata and sets the offset field to -1.
Group Membership API
Possible Error Codes
* UNKNOWN_TOPIC_OR_PARTITION (3) <- only for request v0
* GROUP_LOAD_IN_PROGRESS (14)
* NOT_COORDINATOR_FOR_GROUP (16)
* ILLEGAL_GENERATION (22)
* UNKNOWN_MEMBER_ID (25)
* TOPIC_AUTHORIZATION_FAILED (29)
* GROUP_AUTHORIZATION_FAILED (30)
Group Membership API
These requests are used These requests are used by clients to participate in a client group managed by Kafka. From a high level, each group in the cluster is assigned one the brokers (its group coordinator) to facilitate group management. Once the coordinator has been located (using the group coordinator request from above), group members can join the group and synchronize state, and then use heartbeats to stay active in the group. When the client shuts down, it uses a leave group request to deregister from the group. More detail on protocol semantics is outlined in Kafka Client-side Assignment Proposal.
...
Consumer groups: The coordinator is responsible for choosing a protocol (i.e. partition assignment strategy) compatible across all members. The leader then is the member which actually performs the selected assignment. Multiple assignment strategies can be included in the join group request to support either an upgrade of an existing version or a change to a different assignment strategy.group request to support either an upgrade of an existing version or a change to a different assignment strategy.
Possible Error Codes:
* GROUP_LOAD_IN_PROGRESS (14)
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_GROUP (16)
* INCONSISTENT_GROUP_PROTOCOL (23)
* UNKNOWN_MEMBER_ID (25)
* INVALID_SESSION_TIMEOUT (26)
* GROUP_AUTHORIZATION_FAILED (30)
SyncGroup Request
The sync group request is used by the group leader to assign state (e.g. partition assignments) to all members of the current generation. All members send SyncGroup immediately after joining the group, but only the leader provides the group's assignment.
...
Code Block |
---|
SyncGroupResponse => ErrorCode MemberAssignment ErrorCode => int16 MemberAssignment => bytes => int16 MemberAssignment => bytes |
Possible Error Codes:
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_GROUP (16)
* ILLEGAL_GENERATION (22)
* UNKNOWN_MEMBER_ID (25)
* REBALANCE_IN_PROGRESS (27)
* GROUP_AUTHORIZATION_FAILED (30)
Heartbeat Request
Once a member has joined and synced, it will begin sending periodic heartbeats to keep itself in the group. If not heartbeat has been received by the coordinator with the configured session timeout, the member will be kicked out of the group.
Code Block |
---|
HeartbeatRequest => GroupId GenerationId MemberId GroupId => string GenerationId => int32 MemberId => string |
Heartbeat Response
...
Code Block |
---|
HeartbeatResponse => ErrorCode
ErrorCode => int16 |
Possible Error Codes:
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_GROUP (16)
* ILLEGAL_GENERATION (22)
* UNKNOWN_MEMBER_ID (25)
* REBALANCE_IN_PROGRESS (27)
* GROUP_AUTHORIZATION_FAILED (30)
LeaveGroup Request
To explicitly leave a group, the client can send a leave group request. This is preferred over letting the session timeout expire since it allows the group to rebalance faster, which for the consumer means that less time will elapse before partitions can be reassigned to an active member.
...
Code Block |
---|
LeaveGroupResponse => ErrorCode
ErrorCode => int16 |
Possible Error Codes:
* GROUP_LOAD_IN_PROGRESS (14)
* CONSUMER_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_CONSUMER (16)
* UNKNOWN_CONSUMER_ID (25)
*
...
GROUP_AUTHORIZATION_FAILED (30)
Administrative API
ListGroups Request
...
Code Block |
---|
ListGroupsResponse => ErrorCode Groups ErrorCode => int16 Groups => [GroupId ProtocolType] GroupId => string ProtocolType => string |
Possible Error Codes:
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* AUTHORIZATION_FAILED (29)
DescribeGroups Request
Code Block |
---|
DescribeGroupsRequest => [GroupId] GroupId => string |
...
Code Block |
---|
DescribeGroupsResponse => [ErrorCode GroupId State ProtocolType Protocol Members]
ErrorCode => int16
GroupId => string
State => string
ProtocolType => string
Protocol => string
Members => [MemberId ClientId ClientHost MemberMetadata MemberAssignment]
MemberId => string
ClientId => string
ClientHost => string
MemberMetadata => bytes
MemberAssignment => bytes |
Possible Error Codes:
* GROUP_LOAD_IN_PROGRESS (14)
* GROUP_COORDINATOR_NOT_AVAILABLE (15)
* NOT_COORDINATOR_FOR_GROUP (16)
* AUTHORIZATION_FAILED (29)
Constants
Api Keys
The following are the numeric codes that the ApiKey in the request can take for each of the above request types.
...