Server processes partitions in order they appear in request.
If top level max_bytes parameter is Int.MAX_INT ("no limit"), the request is processed exactly as before.
Otherwise, for each partition except the first one server fetches up to corresponding partition limit max_bytes, but not bigger than remaining response limit.
This algorithm provides following guarantees:
- FetchRequest with top level max_bytes != Int.MAX_INT always always makes progress - if server has message(s), than at least one message is returned irrespective of max_bytes
- FetchRequest response size will not be bigger than max(max_bytes, size of the first message in first partition)
Compatibility, Deprecation, and Migration Plan
New The new fetch request is designed to work properly even if the top level max_bytes is less than the message size. If max_bytes is Int.MAX_INT, new request behaves exactly like old one.
So we can even make this KIP absolutely transparent for users by making setting default for both max.fetch.bytes and replica.fetch.response.max.bytes to Int.MAX_INT.
However, since clients like ReplicaFetcherThread and Java Consumer are ready for new fetch request, we decided to enable We decided to establish the following defaults:
fetch.max.bytes = 50MB
- Together with addition of global response limit deprecate per-partitions limit. Rejected since per-partition limit can be useful for Kafka streams (see mail list discussion).
- Do random partition shuffling on server side. Pros: ensure fairness without client-side modifications. Cons: non-deterministic behaviour on server side; round-robin can be easily implemented on client side.