DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
MDC is empty: The broker, controller, and all client libraries (producer, consumer, admin) do not set any MDC fields. Only Kafka Connect (via KIP-449) sets
connector.context. This meansJsonTemplateLayoutMDC resolvers produce null for every field on every log line.Context is trapped in string prefixes: The
LogContextclass prepends a human-readable prefix like[BrokerServer id=0]or[Producer clientId=p1]to every log message. This context is useful for text logs but is not available as structured fields for log aggregation, filtering, or alerting.No standard field names: Even if operators manually called
MDC.put()in their code, there is no agreed-upon naming convention for context fields across Kafka components. Each deployment would invent its own names.Cross-component correlation is impossible: There is no shared field (like
kafka.node.id) that appears consistently across all log messages from a given broker, making it hard to filter logs by source in multi-tenant or multi-cluster environments.
Prior Art
| KIP | Kafka Version | What it did | Relationship to this KIP |
|---|---|---|---|
| KIP-653 | 4.0 | Upgraded to Log4j 2.x; enabled JsonTemplateLayout | Provides the JSON output capability. This KIP populates the MDC fields that JSON output needs. |
| KIP-449 | 2.3 | Added MDC context (connector.context) to Kafka Connect | Proved the MDC pattern works in Kafka. This KIP extends the same approach to broker, controller, and clients. |
| KIP-673 | 2.8 | Made request/response DEBUG traces emit proper JSON | Different scope (request traces only). Complementary to this KIP. |
| KIP-714 | 3.6 | Client metrics telemetry via OpenTelemetry Protocol | KIP-714 structures metrics; this KIP structures logs. Together they complete the Kafka observability story. |
| KIP-916 | Under Discussion | Adds flow.context MDC key for MirrorMaker 2 | Same MDC pattern, narrow scope (MM2 only). Complementary to this KIP. |
Industry Context
Structured logging with pre-populated context fields is industry standard:
...
When JSON logging is enabled, the following MDC keys appear as top-level JSON fields. These key names become a stable public contract --- log consumers (dashboards, alerts, analysis tools) will depend on them.
| MDC Key | Type | Description | Phase |
|---|---|---|---|
kafka.node.id | string | Broker or controller node ID | 1 |
kafka.component | string | Component name (e.g., BrokerServer, UnifiedLog) | 1 |
kafka.client.id | string | Client ID (producer, consumer, admin) | 1 |
kafka.client.type | string | One of: producer, consumer, admin | 1 |
kafka.group.id | string | Consumer/share group ID | 1 |
kafka.group.instance.id | string | Static group membership instance ID | 1 |
kafka.transactional.id | string | Transactional producer ID | 1 |
kafka.topic | string | Topic name | 1 |
kafka.partition | string | Partition number | 1 |
kafka.cluster.id | string | Kafka cluster ID | 2 (reserved) |
kafka.connector | string | Kafka Connect connector name | 2 (reserved) |
kafka.task.id | string | Kafka Connect task ID | 2 (reserved) |
All keys use the kafka. prefix to avoid collision with user-defined MDC entries. Keys whose MDC value is null are omitted from the JSON output (not emitted as "kafka.topic": null).
...
The JSON template file defines the output schema for structured log events. Every JSON log line contains these fixed fields:
| Field | Source | Description |
|---|---|---|
timestamp | Log4j2 event | ISO-8601 timestamp in UTC |
level | Log4j2 event | Log level name (INFO, WARN, ERROR, etc.) |
logger | Log4j2 event | Logger name (fully qualified class name) |
message | Log4j2 event | The formatted log message (includes the LogContext prefix) |
thread | Log4j2 event | Thread name |
exception | Log4j2 event | Stack trace string (omitted when no exception) |
kafka.* | MDC | All standard MDC keys from the table above (omitted when null) |
This template file is a configuration artifact, not compiled code. Users can customize it by providing their own template via the eventTemplateUri property in the Log4j2 YAML config.
...
4. New Runtime Dependency
| Artifact | Version | Scope |
|---|---|---|
org.apache.logging.log4j:log4j-layout-template-json | Same as log4j-core (currently 2.25.4) | Runtime (only needed when JSON config is active) |
This artifact is ~80KB with no transitive dependencies beyond log4j-core (already a Kafka dependency). It is added to the log4j2Libs and log4jReleaseLibs dependency groups. The minimum Log4j2 version required for JsonTemplateLayout is 2.14.0; Kafka currently ships 2.25.4.
...
A set of standard MDC key names is defined for consistent structured output:
| MDC Key | Description | Set by | Phase |
|---|---|---|---|
kafka.node.id | Broker or controller node ID | BrokerServer, ControllerServer, SharedServer | 1 |
kafka.component | Component name | Each component | 1 |
kafka.client.id | Client ID | KafkaProducer, KafkaConsumer, KafkaAdminClient | 1 |
kafka.client.type | producer, consumer, or admin | Client constructors | 1 |
kafka.group.id | Consumer group ID | KafkaConsumer, ShareConsumer | 1 |
kafka.group.instance.id | Static group membership ID | KafkaConsumer | 1 |
kafka.transactional.id | Transactional producer ID | KafkaProducer | 1 |
kafka.topic | Topic name | UnifiedLog | 1 |
kafka.partition | Partition number | UnifiedLog | 1 |
kafka.cluster.id | Kafka cluster ID | (Phase 2) | 2 |
kafka.connector | Connect connector name | (Phase 2) | 2 |
kafka.task.id | Connect task ID | (Phase 2) | 2 |
All key names use the kafka. prefix to avoid collision with user-defined MDC entries.
...
Three new Log4j2 configuration files are provided alongside the existing text configs:
| File | Purpose | Replaces |
|---|---|---|
config/log4j2-json.yaml | Broker/Controller JSON logging | config/log4j2.yaml |
config/connect-log4j2-json.yaml | Connect worker JSON logging | config/connect-log4j2.yaml |
config/tools-log4j2-json.yaml | CLI tools JSON logging | config/tools-log4j2.yaml |
These configs use Log4j2's JsonTemplateLayout with a shared template file (config/log4j2-json-template.json) that defines the output schema.
...