Table of Contents |
---|
Status
Current state: Under Discussion Accepted
Discussion thread: here
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Allowing remote storage plugins to write custom metadata of arbitrary size is not good for the stability of the cluster. The broker should have a configurable limit for their size. If a piece of custom metadata exceeds the limit, the execution of RLMTask.copyLogSegmentsToRemote
should be interrupted with an error message. The default value for the limit should be 10 KiB, which is enough to store 10 1024-byte ANSI strings (minus some bytes for e.g. the map structure), which is quite sufficient for the two use cases from the Motivation part and on the other hand 128 B, which is small enough to not be disruptive.
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "name": "CustomMetadata", "type": "bytes", "default": "null", "versions": "0+", "nullableVersions": "0+", "about": "Custom metadata." } |
RemoteLogSegmentMetadataUpdateRecord
Same as for RemoteLogSegmentMetadataRecord
.
RemoteLogSegmentMetadataSnapshotRecord
...
Key Name | Description | Valid Values | Default Value |
---|---|---|---|
remote.log.metadata.custom.metadata.max.sizebytes | The maximum size of custom metadata in bytes that the broker should accept from a remote storage plugin. If custom metadata custom metadata exceeds this limit, the whole updated segment metadata will not be stored, the copied data will be attempted to delete, and the remote copying task for this topic-partition will stop with an error. | 0..Integer.MAX_VALUE |
|
Compatibility, Deprecation, and Migration Plan
...