Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Motivation

Apache Kafka is widely used for asynchronous communication between services, with Protocol Buffers being a common serialization format due to its efficiency and strong typing. In production, schema registries provide visibility into message formats. However, during local development and debugging, engineers often operate outside this managed ecosystem — schema registry access may be unavailable, and protobuf messages are not human-readable in binary form.

This forces developers to write ad hoc deserialization scripts, increasing debugging time and cognitive overhead. There is no lightweight, built-in solution for local protobuf inspection without schema registry dependency.

Public Interfaces

New class: org.apache.kafka.tools.consumer.ProtobufMessageFormatter implementing MessageFormatter. No existing interfaces are modified

Proposed Changes

This proposal adds a ProtobufMessageFormatter to Kafka's tools module, implementing the existing org.apache.kafka.common.MessageFormatter interface. This extends Kafka's existing formatter model (already used by string and JSON formatters) to support Protocol Buffers natively from the console consumer.

Key behaviors:

  • Loads .proto definitions dynamically from a user-specified directory at runtime
  • Deserializes binary message values into human-readable JSON-like output
  • Requires no schema registry integration
  • Integrates with existing CLI tooling via --formatter and --property:


bin/kafka-console-consumer.sh \
  --topic <topic> \
  --bootstrap-server <server> \
  --formatter org.apache.kafka.tools.consumer.ProtobufMessageFormatter \
  --property proto.dir=/path/to/protos \
  --property message.type=com.example.MyEvent

Compatibility, Deprecation, and Migration Plan

Fully backward compatible. This is an additive change only — a new optional formatter with no effect on existing functionality. Introduces a new dependency on protobuf-java scoped to the tools module.

Test Plan

  • Unit tests covering successful deserialization of known message types
  • Tests for missing or malformed .proto files (graceful error handling)
  • Tests for unknown message types
  • Integration test via kafka-console-consumer.sh with a sample protobuf-encoded topic

Rejected Alternatives

  • Adding a new Kafka CLI tool
  • No labels