Status

Current state: Accepted

Discussion thread: here

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The  `EndToEndLatency` tool is a crucial utility for measuring data transmission latency from producers to brokers and then to consumers. It provides essential performance metrics for all Kafka users. However, the current implementation suffers from several limitations:

...

This KIP aims to address these limitations.

Public Interfaces

CLI

The usage of the kafka-e2e-latency.sh  script will be updated.

kafka-e2e-latency.sh

Current version with fixed array indices:

...

• --bootstrap-server
• --topic
• --num-records
• --producer-acks
• --record-size
• --record-key-size [optional]
• --num-headers [optional]
• --record-header-key-size [optional]
• --record-header-size [optional]
• --command-config [optional]

Proposed Changes

The kafka-e2e-latency.sh  tool will be enhanced with the following improvements:

...

• broker_list → bootstrap-server

• num_messages → num-records

• message_size_bytes → record-size

• properties_file → command-config

Usage

order-independent arguments with an argument parser

...

record-header-size (128) : The size of header values per message in bytes.

Compatibility, Deprecation, and Migration Plan

This change is designed to be fully backward compatible:

...

The previous syntax using positional arguments is deprecated and will be removed from the parser in AK 5.0. A deprecation warning message will be displayed if positional arguments are used to encourage users to move to the new syntax.

Test Plan

New and updated test cases will be added to `EndToEndLatencyTest.java` to ensure the correct functionality and compatibility of the changes:

• Key and Header Testing: Test cases will be added to validate the correct behavior when `--record-key-size`,`--num-headers`,`--record-header-key-size` and  `--record-header-size` are specified, including scenarios with various size values and combinations.
• Argument Parser Validation: Comprehensive tests will be implemented to ensure the updated argument parser correctly handles all existing and new options, including default values, error conditions (e.g., missing required arguments), and proper parsing of all data types.
• Backward Compatibility: Tests will confirm that the tool continues to function correctly when `--record-key-size`,`--num-headers`,`--record-header-key-size` and `--record-header-size` are omitted, mimicking the current behavior.

Rejected Alternatives

N/A