Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


KIP round-up

Next KIP Number: 903904

Use this number as the identifier for your KIP and increment this value.


KIP-59: Proposal for a kafka broker commandSent emails to Dev discussion group. Work tracked under KAFKA-3663.
KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback
KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user
KIP 141 - ProducerRecord & SourceRecord: Add timestamp constructors
KIP-142: Add ListTopicsRequest to efficiently list all the topics in a cluster
KIP-148: Add a connect timeout for client
KIP-166 - Add a tool to make amounts of replicas and leaders on brokers balanced
KIP-169 - Lag-Aware Partition Assignment Strategy
KIP-178: Size-based log directory selection strategy

KIP-185: Make exactly once in order delivery the default producer setting

KIP-193: Add SchemaBuilder.from(Schema)
KIP-199: Add Kafka Connect offset tool
KIP-201: Rationalising Policy interfaces

KIP-209 - Connection String Support

KIP-217: Expose a timeout to allow an expired ZK session to be re-created
KIP-228 Negative record timestamp supportVoting in progress
KIP-234: add support for getting topic defaults from AdminClient
KIP-236: Interruptible Partition Reassignment Discussion
KIP-240: AdminClient.listReassignments() AdminClient.describeReassignments()
KIP-242: Mask password in Kafka Connect Rest API response
KIP-250 Add Support for Quorum-based Producer Acknowledgment
KIP-252 - Extend ACLs to allow filtering based on ip ranges and subnets
KIP-253: Support in-order message delivery with partition expansion Discussion
KIP-254: JsonConverter Exception Handeling
KIP-264: Add a consumer metric to record raw fetch sizeVoting in progress
KIP-271: Add NetworkClient redirectorDiscussion
KIP-273: Kafka to support using ETCD beside ZookeeperDiscussion
KIP-293 Add new metrics for consumer/replication fetch requestsVoting in progress
KIP-296: Add connector level configurability for producer/consumer client configs Discussion
KIP-301: Schema Inferencing for JsonConverterDiscussion
KIP-304: Connect runtime mode improvements for container platformsDiscussion
KIP-315: Stream Join Sticky AssignorDiscussion
KIP-316: Command-line overrides for ConnectDistributed worker propertiesDiscussion
KIP-317: Add end-to-end data encryption functionality to Apache KafkaDiscussion
KIP-325: Extend Consumer Group Command to Show Beginning OffsetsVoting in Progress345
KIP-333: Add faster mode of rebalancingDiscussion
KIP-334 - Include partitions in exceptions raised during consumer record deserialization/validation
KIP-348 Eliminate null from SourceTask#poll()
KIP-350: Allow to take brokerid as parameter to show partitions associated with it
KIP-356: Add KafkaConsumer fetch-error-rate and fetch-error-total metricsDiscussion
KIP-363: Allow performance tools to print final results to output fileDiscussion
KIP-370: Remove Orphan PartitionsDiscussion
KIP-375: Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurableDiscussion
KIP-381: Connect: Tell about records that had their offsets flushed in callback

Voting in progress (restarted 18th January 2019, due to no votes in first attempt)

KIP-384: Add config for incompatible changes to persistent metadataDiscussion
KIP-385: Avoid throwing away prefetched dataDiscussion
KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer


KIP-388: Add observer interface to record request and responseDiscussion
KIP-395: Encypt-then-MAC Delegation token metadata
KIP-397: Add methods to override fetch offsets based on timestampDiscussion
KIP-400: Improve exit status in case of errors in ConsoleProducerDiscussion
KIP-403: Increase ProducerPerformance precision by using nanoTimeDraft
KIP-406: GlobalStreamThread should honor custom offset policy.Blocked
KIP-407: Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api

KIP-409: Allow creating under-replicated topics and partitions

KIP-410: Add metric for request handler thread pool utilization by request type

KIP-416: Notify SourceTask of ACK'd offsets, metadataDiscussion
KIP-419: Safely notify Kafka Connect SourceTask is stoppedVoting in progress, JIRA exists with pull request
KIP-423: Add JoinReason to Consumer Join Group Protocol


KIP-424: Allow suppression of intermediate events based on wall clock timeDiscussion
KIP-426: Persist Broker Id to ZookeeperDiscussion
KIP-435: Internal Partition Reassignment BatchingDiscussion
KIP-438: Expose task, connector IDs in Connect APIDiscussion
KIP-439: Cleanup built-in Store interfacesDiscussion
KIP-448: Add State Stores Unit Test Support to Kafka Streams Test UtilsDiscussion
KIP-452: Tool to view cluster statusDiscussion, JIRA exists
KIP-468: Avoid decompression of record when validate record at server in the scene of inPlaceAssignment .Under discussion
KIP-473: Enable KafkaLog4JAppender to use SASL Authentication Callback HandlersUnder Discussion 
KIP-477: Add PATCH method for connector config in Connect REST APIUnder Discussion
KIP-487: Automatic Topic Creation on ProducerUnder Discussion
KIP-489: Kafka Consumer Record Latency MetricUnder Discussion
KIP-490: New metric to count offsets expired without being consumed by a consumer groupUnder discussion
KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)Under discussion
KIP-494: Connect REST Endpoint to get Transformations (SMTs)
KIP-498: Add client-side configuration for maximum response size to protect against OOMSent emails to Dev discussion group. Work tracked under KAFKA-4090.

KIP-499 - Unify connection name flag for command line tool

Under discussion
KIP-502: Connect SinkTask.put(...) to specify ArrayList<SinkRecord> in Signature
KIP-505: Add new public method to only update assignment metadata in consumerUnder discussion
KIP-506: Allow setting SCRAM password via Admin interfaceUnder discussion
KIP-508: Make Suppression State QueriableUnder discussion
KIP-509: Rebalance and restart ProducersUnder discussion
KIP-510: Metrics library upgradeUnder discussion
KIP-512: Adding headers to RecordMetaDataSent mail to dev discussion group
KIP-513: Distinguish between Key and Value serdes in scala wrapper library for kafka streamsSent mail to dev discussion group
KIP-536: Propagate broker start time to Admin APIUnder discussion
KIP-539: Add mechanism to flush records out in low volume suppression buffersUnder discussion.
KIP-540: Implement per key stream time trackingDraft (In Progress)
KIP-542: Partition Reassignment ThrottlingDraft
KIP-547: Extend ConsumerInterceptor to allow modification of Consumer CommitsUnder Discussion
KIP-548 Add Option to enforce rack-aware custom partition reassignment executionUnder Discussion
KIP-549: Surface Consumer's Metadata in KafkaAdminClient#describeConsumerGroupsUnder Discussion
KIP-550: Mechanism to Delete Stray Partitions on BrokerDraft
KIP-552: Suppress "unused config" warning

Under Discussion

KIP-560: Auto infer external topic partitions in stream reset tool

Under Discussion

KIP-561: Regex Support for ConsumerGroupCommand

Under Discussion
KIP-563: Add 'tail -n' feature for ConsoleConsumerUnder Discussion
KIP-564: Add new cached authorizer:change the dim of cacheUnder Discussion
KIP-565: Using AclCommand,avoid call the global method loadcache in SimpleAclAuthorizerUnder Discussion
KIP-566: Add rebalance callbacks to ConsumerInterceptorUnder Discussion
KIP-567: Kafka Cluster Audit

Under Discussion

KIP-575: build a Kafka-Exporter by JavaUnder Discussion
KIP-576: Support dynamic update of more broker configs related to replicationUnder Discussion
KIP-578: Add configuration to limit number of partitionsVoting
KIP-579: new exception on min.insync.replicas > replication.factorVoting in progress
KIP-581: Value of optional null field which has default valueUnder Discussion
KIP-582: Add a "continue" option for Kafka Connect error handlingUnder Discussion
KIP-583: add tag "partition" to BrokerTopicMetrics so as to observe the partition metrics on the same brokerUnder Discussion
KIP-587: Suppress detailed responses for handled exceptions in security-sensitive environmentsUnder Discussion
KIP-592: Replicate mirrormaker topics from earliestUnder Discussion
KIP-595: A Raft Protocol for the Metadata QuorumUnder Discussion
KIP-596: Safely abort Producer transactions during application shutdownUnder Discussion
KIP-598: Augment TopologyDescription with store and source / sink serde informationUnder Discussion
KIP-619: Add internal topic creation supportUnder Discussion
KIP-625: Richer encodings for integral-typed protocol fieldsUnder Discussion

KIP-628: ConsumerPerformance's multi-thread implementation

Sent email to Dev discussion group. Work tracked under KAFKA-10136.
KIP-636: Make RPC error codes and messages tagged fieldsUnder Discussion
KIP-637: Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning eventsDraft
KIP-639: Move nodeLevelSensor and storeLevelSensor methods from StreamsMetricsImpl to StreamsMetricsUnder Discussion
KIP-640: Add log compression analysis toolUnder Discussion
KIP-641 An new java interface to replace 'kafka.common.MessageReader'Under Discussion
KIP-649: Dynamic client configurationUnder Discussion
KIP-655: Windowed Distinct Operation for Kafka Streams APIVoting in progress
KIP-656: MirrorMaker2 Exactly-once SemanticsDraft
KIP-665: Kafka Connect Hash SMT

Voting in progress

KIP-668: Expose REST endpoint to list converter pluginsUnder Discussion
KIP-669 Preserve Source Partition in Kafka Streams from contextDraft

KIP-674: API to Aggregate Metrics in Kafka Streams

Under Discussion

KIP-675: Convert a KTable to a KStream using the previous valueUnder Discussion
KIP-678: New Kafka Connect SMT for plainText => struct with RegexVoting
KIP-681: Rename master key in delegation token featureDraft
KIP-682: Connect TimestampConverter support for multiple fields and multiple input formatsUnder Discussion
KIP-683: Add recursive support to Connect Cast and ReplaceField transforms, and support for casting complex types to either a native or JSON stringUnder Discussion
KIP-685: Loosen permission for listing reassignmentsUnder Discussion
KIP-686: API to ensure Records policy on the brokerUnder Discussion
KIP-687: Automatic Reloading of Security StoreUnder Discussion
KIP-688: Support dynamic update of delete.topic.enable config

Under Discussion

KIP-692: Make AdminClient value object constructors publicUnder discussion

KIP-693: Client-side Circuit Breaker for Partition Write Errors

Voting in progress

KIP-694: Support Reducing Partitions for Topics

Under Discussion
KIP-697: Stricter parsing of addresses in configsUnder Discussion
KIP-701: Add --override option to bin/connect-distributed.shUnder Discussion
KIP-702: The control plane needs to force the validation of requests from the controllerUnder Discussion
KIP-703: Add a metric for reporting idle connections closed

Under Discussion

KIP-705: Selectively Disable Topology OptimizationsDraft
KIP-706: Add method "Producer#produce" to return CompletionStage instead of FutureUnder Discussion
KIP-711 Deprecate org.apache.kafka.streams.errors.BrokerNotFoundExceptionUnder Discussion
KIP-712: Shallow MirroringUnder Discussion
KIP-713: Validation of Enums in configurationUnder Discussion

KIP-714: Client metrics and observability

Under Discussion
KIP-718: Make KTable Join on Foreign key unopinionatedUnder Discussion
KIP-723: Add property to Kafka ConfigUnder Discussion

KIP-727: Add --under-preferred-replica-partitions option to describe topics command

Under Discussion

KIP-729: Custom validation of records on the broker prior to log append

Under Discussion
KIP-731: Record Rate Limiting for Kafka ConnectUnder Discussion
KIP-736: Report the true end to end fetch latencyUnder Discussion
KIP-737: Add canTrackSource to ReplicationPolicyUnder Discussion
KIP-739: Block Less on KafkaProducer#sendUnder Discussion
KIP-749: Add --files and --file-separator options to the ConsoleProducerUnder Discussion
KIP-752: Support --bootstrap-server in ReplicaVerificationToolUnder Discussion
KIP-754: Make Scala case class's finalUnder Discussion
KIP-755: Add new AUTO_CREATE ACL for auto topic creationUnder Discussion
KIP-756: Move StreamsResetter tool outside of coreUnder Discussion
KIP-759: Unneeded repartition cancelingUnder Discussion
KIP-760: Minimum value for and segment.bytesUnder Discussion
KIP-762: Delete Committed Connect RecordsUnder Discussion
KIP-767: Connect Latency MetricsUnder Discussion
KIP-774: Deprecate public access to Admin client's *Result constructors

Under Discussion

KIP-777: Improved testability for Admin clientDraft
KIP-780: Support fine-grained compression optionsUnder Discussion
KIP-781: Improve MirrorMaker2's client configurationUnder Discussion
KIP-782: Expandable batch size in producerVoting
KIP-785: Automatic storage formattingUnder Discussion
KIP-786: Emit Metric Client Quota ValuesUnder Discussion 
KIP-789: Use localhost:9092 as default bootstrap-server/broker-list in client toolsKIP page WIP
KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)Under Discussion
KIP-794: Strictly Uniform Sticky PartitionerAccepted
KIP-799: Align behaviour for producer callbacks with documented behaviourUnder Discussion

KIP-802: Validation Support for Kafka Connect SMT Options

Under Discussion

KIP-803: Add Task ID and Connector Name to Connect Task ContextUnder Discussion
KIP-804: OfflinePartitionsCount Tagged by TopicUnder Discussion
KIP-807: Refactor KafkaStreams exposed metadata hierarchyUnder Discussion
KIP-809: Support live upgrades with dynamic addition/removal of modular topologiesKIP page WIP
KIP-816: Topology changes without local state resetUnder Discussion
KIP-817: Fix inconsistency in dynamic application log levelsUnder Discussion
KIP-819: Merge multiple KStreams in one operationUnder Discussion
KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocolsUnder Discussion
KIP-823: Update Admin::describeConfigs to allow fetching specific configurationsKIP page WIP
KIP-824: Allowing dumping segmentlogs limiting the batches in the outputAccepted
KIP-825: introduce a new API to control when aggregated results are producedAccepted

KIP-826: Define platforms supported

Under Discussion
KIP-828: Add the corresponding validator to the configuration where the validator is missingUnder Discussion
KIP-829: (console-consumer) add print.topic propertyUnder Discussion
KIP-838 Simulate batching and compressionDiscussion

KIP-839: Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

Under Discussion
KIP-842: Add richer group offset reset mechanismsVoting
KIP-849: Expose logdirs total and usable space via kafka-log-dirs.shUnder Discussion
KIP-852: Optimize calculation of size for log in remote tierDiscussion
KIP-853: KRaft Voter ChangesUnder Discussion
KIP-855: Add schema.namespace parameter to SetSchemaMetadata SMT in Kafka ConnectUnder Discussion
KIP-856: KRaft Disk Failure RecoveryDraft
KIP-857: Streaming recursion in Kafka StreamsUnder Discussion
KIP-858: Handle JBOD broker disk failure in KRaftUnder Discussion

KIP-860: Add client-provided option to guard against replication factor change during partition reassignments

Under Discussion
KIP-864: Add End-To-End Latency Metrics to ConnectorsUnder Discussion
KIP-870: Retention policy based on record event timeUnder Discussion

KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

Under Discussion
KIP-873: ExceptionHandlingDeserializer, RetryDeserializer, PipeSerializer, PipeDeserializerUnder Discussion
KIP-875: First-class offsets support in Kafka ConnectVoting in progress

KIP-877: Mechanism for plugins and connectors to register metrics

Under Discussion
KIP-878: Autoscaling for Stateless & Statically Partitioned StreamsUnder Discussion
KIP-879: Multi-level Rack AwarenessUnder Discussion
KIP-880: X509 SAN based SPIFFE URI ACL within mTLS Client CertificatesUnder Discussion
KIP-882: Kafka Connect REST API timeout improvementsUnder Discussion

KIP-883: Add isDeleted flag when stopping a connector

Under Discussion
KIP-885: Expose Broker's Name and Version to ClientsUnder Discussion
KIP-886: Add Client Producer and Consumer BuildersUnder Discussion

KIP-888: Batch describe ACLs and describe client quotas


KIP-890: Transactions Server-Side Defense

Under Discussion

KIP-891: Running multiple versions of a connectorUnder Discussion
KIP-894: Use incrementalAlterConfig for syncing topic configurationsVoting in progress
KIP-895: Dynamically refresh partition count of __consumer_offsetsUnder Discussion
KIP-896: Remove old client protocol API versions in Kafka 4.0Under Discussion
KIP-897: Publish a single kafka (aka core) Maven artifact in Apache Kafka 4.0Draft
KIP-898: Modernize Connect plugin discoveryUnder Discussion
KIP-899: Allow clients to rebootstrap

Under Discussion

KIP-900: KRaft API additions to support SCRAM for Kafka Brokers

Under Discussion

KIP-901: Add connectorDeleted flag when stopping tasks

Under Discussion

KIP-902: Upgrade Zookeeper to 3.8.1

Under Discussion

KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

Under Discussion

Dormant/inactive KIPs