Status

Current state: Under Discussion

Discussion thread: 

JIRA:

Released: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In KIP-827 we exposed logdirs total and usable space via Kafka API, we can also expose this through kafka-log-dirs.sh.

Public Interfaces

Currently, we output the logDir info in the form of a JSON, here is an example:


{
    "version":1,
    "brokers":[
        {
            "broker":1,
            "logDirs":[
                {
                    "logDir":"/tmp/kraft-combined-logs",
                    "error":null,
                    "partitions":[
                        {
                            "partition":"foo-0",
                            "size":0,
                            "offsetLag":0,
                            "isFuture":false
                        }
                    ]
                }
            ]
        }
    ]
}


we are just adding totalBytes and usableBytes to every json entity in $.brokers.logDirs, and bumping the $.version to 2.

Proposed Changes

We already have a LogDirsCommand which can be used to get the logdirs and details about the replicas they contain, here is the demo:

bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --topic-list foo --describe

Querying brokers for log directories information
Received log directory information from brokers 1
{
    "version":1,
    "brokers":[
        {
            "broker":1,
            "logDirs":[
                {
                    "logDir":"/tmp/kraft-combined-logs",
                    "error":null,
                    "partitions":[
                        {
                            "partition":"foo-0",
                            "size":0,
                            "offsetLag":0,
                            "isFuture":false
                        }
                    ]
                }
            ]
        }
    ]
}


we are just adding totalBytes and usableBytes to every json entity in logDirs, and set version=2, here is an demo:

bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --topic-list foo --describe

Querying brokers for log directories information
Received log directory information from brokers 1
{
    "version":2,
    "brokers":[
        {
            "broker":1,
            "logDirs":[
                {
                    "logDir":"/tmp/kraft-combined-logs",
                    "totalBytes":{
                        "empty":false,
                        "present":true,
                        "asLong":245107195904
                    },
                    "error":null,
                    "usableBytes":{
                        "empty":false,
                        "present":true,
                        "asLong":16010731520
                    },
                    "partitions":[
                        {
                            "partition":"foo-0",
                            "size":0,
                            "offsetLag":0,
                            "isFuture":false
                        }
                    ]
                }
            ]
        }
    ]
}


Compatibility, Deprecation, and Migration Plan

This change only added 2 fields to JSON schema so it's expected to be compatible.

Test Plan

Test locally using a local Kafka cluster.

Rejected Alternatives


  • No labels