Status

Current state: Accepted

Discussion thread: https://lists.apache.org/thread/dvc6pp2h5oxo4bcq1mnfbs02z0g7pk61

Voting thread: https://lists.apache.org/thread/nqkv6ckppz5o6s9h1ls3qzhlmwchxqn1

JIRA: KAFKA-12392

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The ConsoleProducer has two options, --batch-size and --max-partition-memory-bytes, both of which configure the underlying batch.size producer config. One of them should be removed.

A previously discarded KIP-717 suggested removing --batch-size. However, this KIP argues for the opposite. Retaining --batch-size is more straightforward and aligns better with the terminology used throughout the Kafka community.

Public Interfaces

This change affects the command-line interface of 

  • kafka-console-producer.sh
    • Using the --max-partition-memory-bytes option will display a warning:
      "Warning: --max-partition-memory-bytes is deprecated and will be removed in a future version. Use --batch-size instead." 

Proposed Changes

  • Correct the inaccurate description of --batch-size to reflect that it controls byte size, not message count
  • Mark --max-partition-memory-bytes as deprecated, directing users to --batch-size instead.

  • To maintain backward compatibility, the existing behavior where --max-partition-memory-bytes overrides --batch-size will be preserved until the deprecated option is removed.

Before:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             Number of messages to send in a single 
                                           batch if they are not being sent     
                                           synchronously. please note that this 
                                           option will be replaced if max-      
                                           partition-memory-bytes is also set   
                                           (default: 16384)                     
...

--max-partition-memory-bytes <Integer:   The buffer size allocated for a        
  memory in bytes per partition>           partition. When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)

After:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             The buffer size in bytes allocated for a partition.
 										   When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)
             							   please note that this option will be 
										   replaced if max-partition-memory-bytes
										   is also set.

--max-partition-memory-bytes <Integer:   (Deprecated) The buffer size allocated for a partition.
  memory in bytes per partition>           When records are received
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)
							         	   This option will be removed in a 
										   future version. Use --batch-size instead.

./kafka-console-producer.sh --topic test --bootstrap-server localhost:9092 --max-partition-memory-bytes 16384
Warning: --max-partition-memory-bytes is deprecated and will be removed in a future version. Use --batch-size instead.

Compatibility, Deprecation, and Migration Plan

  • Users should replace --max-partition-memory-bytes with --batch-size in their scripts.
  • The deprecated option is planned for complete removal in the next major release

Test Plan

N/A

Rejected Alternatives

Deprecate the --batch-size option

Use --max-memory-bytes to indicate batch.size in ProducerConfig is less intuitive.

Deprecate both --batch-size and --max-memory-bytes in favor of generic properties

Encouraging users to set producer properties via the --command-config  or --command-property  options.

This was rejected because keeping the dedicated --batch-size flag is more convenient for users who frequently test the interaction between batch.size  and linger.ms.

Note: The recent KIP-1147 unified option names across many tools. It replaced --producer-property and --consumer-property with a consistent --command-property, and unified file-based options into --command-config.

  • No labels