This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Kafka Improvement Proposals
535 more child pages

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


KIP-59: Proposal for a kafka broker commandSent emails to Dev discussion group. Work tracked under KAFKA-3663.
KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback 
KIP-131 - Add access to OffsetStorageReader from SourceConnector 
KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user 
KIP 141 - ProducerRecord & SourceRecord: Add timestamp constructors 
KIP-142: Add ListTopicsRequest to efficiently list all the topics in a cluster 
KIP-148: Add a connect timeout for client 
KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics 
KIP-159: Introducing Rich functions to Streams 
KIP-166 - Add a tool to make amounts of replicas and leaders on brokers balanced 
KIP-169 - Lag-Aware Partition Assignment Strategy 
KIP-178: Size-based log directory selection strategy


KIP-185: Make exactly once in order delivery the default producer setting

KIP-193: Add SchemaBuilder.from(Schema) 
KIP-199: Add Kafka Connect offset tool 
KIP-201: Rationalising Policy interfaces 
KIP-209: Connection String Support 
KIP-213 Support non-key joining in KTable 
KIP-216: IQ should throw different exceptions for different errors 
KIP-217: Expose a timeout to allow an expired ZK session to be re-created 
KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint 
KIP-228 Negative record timestamp supportVoting in progress
KIP-234: add support for getting topic defaults from AdminClient 
KIP-236: Interruptible Partition Reassignment 
KIP-240: AdminClient.listReassignments() AdminClient.describeReassignments() 
KIP-242: Mask password in Kafka Connect Rest API response 
KIP-248 - Create New ConfigCommand That Uses The New AdminClientVoting in progress (pending on more binding votes)
KIP-250 Add Support for Quorum-based Producer Acknowledgment 
KIP-252 - Extend ACLs to allow filtering based on ip ranges and subnets 
KIP-253: Support in-order message delivery with partition expansion Discussion
KIP-254: JsonConverter Exception Handeling 
KIP-259: Improve Streams DSL Timestamp Propagation Semantics 
KIP-260: add primary join operation for Stream-Stream join (WIP)Draft
KIP-264: Add a consumer metric to record raw fetch sizeVoting in progress
KIP-271: Add NetworkClient redirectorDiscussion
KIP-273: Kafka to support using ETCD beside ZookeeperDiscussion
KIP-275 - Indicate "isClosing" in the SinkTaskContextVoting in progress
KIP-280: Enhanced log compactionDiscussion

KIP-291: Separating controller connections and requests from the data plane

KIP-293 Add new metrics for consumer/replication fetch requestsVoting in progress
KIP-296: Add connector level configurability for producer/consumer client configs Discussion
KIP-300: Add Windowed KTable API in StreamsBuilderDiscussion
KIP-301: Schema Inferencing for JsonConverterDiscussion
KIP-304: Connect runtime mode improvements for container platformsDiscussion
KIP-307: Allow to define custom processor names with KStreams DSLDiscussion
KIP-314: KTable to GlobalKTable Bi-directional JoinDiscussion
KIP-315: Stream Join Sticky AssignorDiscussion
KIP-316: Command-line overrides for ConnectDistributed worker propertiesDiscussion
KIP-317: Add transparent data encryption functionalityDiscussion
KIP-320: Allow fetchers to detect and handle log truncationDiscussion
KIP-325: Extend Consumer Group Command to Show Beginning OffsetsVoting in Progress
KIP-326: Schedulable KTable as Graph sourceDiscussion
KIP-333: Add faster mode of rebalancingDiscussion
KIP-334 - Include partitions in exceptions raised during consumer record deserialization/validation
KIP-335: Consider configurations for KafkaStreams


KIP-345: Introduce static membership protocol to reduce consumer rebalancesAccepted
KIP-347: Enable batching in FindCoordinatorRequestDiscussion
KIP-348 Eliminate null from SourceTask#poll()
KIP-349: Priorities for Source TopicsDiscussionKIP-350: Allow to take brokerid as parameter to show partitions associated with it
KIP-354: Add a Maximum Log Compaction LagAccepted
KIP-356: Add KafkaConsumer fetch-error-rate and fetch-error-total metricsDiscussion
KIP-360: Improve handling of unknown producerDiscussion
KIP-362: Support Dynamic Session WindowDiscussion
KIP-363: Allow performance tools to print final results to output fileDiscussion
KIP-369: Alternative Partitioner to Support "Always Round-Robin" SelectionDiscussion
KIP-370: Remove Orphan PartitionsDiscussion
KIP-373: Allow users to create delegation tokens for other usersDiscussion
KIP-374: Add '--help' option to all available Kafka CLI commandsAccepted
KIP-375: Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurableDiscussion
KIP-378: Enable Dependency Injection for Kafka Streams handlersDiscussion

KIP-379: Multiple Consumer Group Management

KIP-381: Connect: Tell about records that had their offsets flushed in callback

Voting in progress (restarted 18th January 2019, due to no votes in first attempt)

KIP-383:  Pluggable interface for SSL FactoryVoting in progress
KIP-384: Add config for incompatible changes to persistent metadataDiscussion
KIP-385: Avoid throwing away prefetched dataDiscussion
KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer


KIP-388: Add observer interface to record request and responseDiscussion
KIP-389: Introduce a configurable consumer group size limitAccepted
KIP-390: Allow fine-grained configuration for compressionDiscussion, JIRA exists with pull-request
KIP-391: Allow Producing with Offsets for Cluster ReplicationDiscussion, JIRA exists with pull-request
KIP-392: Allow consumers to fetch from closest replicaDiscussion
KIP-395: Encypt-then-MAC Delegation token metadata

KIP-396: Add Commit/List Offsets Operations to AdminClient

KIP-397: Add methods to override fetch offsets based on timestampDiscussion
KIP-398: Support reading trust store from classpathDiscussion
KIP-399: Extend ProductionExceptionHandler to cover serialization exceptionsDiscussion
KIP-400: Improve exit status in case of errors in ConsoleProducerDiscussion
KIP-401: TransformerSupplier/ProcessorSupplier EnhancementsDiscussion
KIP-403: Increase ProducerPerformance precision by using nanoTimeDraft
KIP-405: Kafka Tiered Storage


KIP-406: GlobalStreamThread should honor custom offset policy.Discussion
KIP-407: Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api

KIP-409: Allow creating under-replicated topics and partitions


KIP-410: Add metric for request handler thread pool utilization by request type

KIP-408: Add asynchronous processing to Kafka StreamsDiscussion

KIP-411: Add option to make Kafka Connect task client ID values unique

KIP-412: Extend Admin API to support dynamic application log levelsDiscussion
KIP-415: Incremental Cooperative Rebalancing in Kafka ConnectDiscussion
KIP-416: Notify SourceTask of ACK'd offsets, metadataDiscussion
KIP-417: Allow JmxTool to connect to a secured RMI portDiscussion, JIRA exists with pull-request
KIP-418: A method-chaining way to branch KStreamDiscussion, JIRA exists with pull-request
KIP-419: Safely notify Kafka Connect SourceTask is stoppedDiscussion
KIP-420: Add Single Value Fetch in Session StoresVoting
KIP-421: Support resolving externalized secrets in AbstractConfigDiscussion
KIP-422: Add support for user/client configuration in the Kafka Admin ClientDiscussion, JIRA exists with pull-request


Discarded KIPs

KIP-5 - Broker Configuration ManagementSuperseded by KIP-21
KIP-7 - Security - IP Filtering 
KIP-9 - SSL SupportAdopted via KAFKA-1690 but not via this KIP process
KIP-18 - JBOD SupportSuperseded by KIP-112 and KIP-113 
KIP-24 - Remove ISR information from TopicMetadataRequest and add broker level metadata request 
KIP-29 - Add an IsrPropagateIntervalMs configuration to KafkaConfigNo longer need after KAFKA-2722

KIP-34 Add Partitioner Change Listener to Partitioner Interface for Multiple Use Case

Per feedback it is better do this parition() method and avoid Thread Coordination etc.
KIP-80: Kafka Rest Server 
KIP-83 - Allow multiple SASL authenticated Java clients in a single JVM processAfter KIP-85, no interface changes, reduced to KAFKA-4180
KIP-116: Add State Store Checkpoint Interval ConfigurationCurrently not needed as checkpointing can be done on commit interval.
KIP-127: Pluggable JAAS LoginModule configuration for SSLSimilar functionality can be implemented using a custom PrincipalBuilder.
KIP-132: Augment KStream.print to allow extra parameters in the printed string

Duplicated by KIP-160

KIP-184: Rename LogCleaner and related classes to LogCompactorA lot of configuration change will cause more trouble and doesn't seems to be worth it
KIP-111: Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.Covered by KIP-189
KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methodsCovered by KIP-182
KIP-165: Extend Interactive Queries for return latest update timestamp per keyCovered by KIP-258
KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicySuperseded by KIP-201: Rationalising Policy interfaces
KIP 172: Add regular-expression topic support for sink connectorSuperseded by KIP-215: Add topic regex support for Connect sinks
KIP-179: Change ReassignPartitionsCommand to use AdminClient
KIP 230: Name Windowing JoinsCovered by KIP-372: Naming Repartition Topics for Joins and Grouping
KIP-232: Detect outdated metadata using per-partition leaderEpoch fieldMerged into KIP-320: Allow fetchers to detect and handle log truncation
KIP-246: Connect producers and consumers should inherit worker configsWithdrawn because proposal could not be made backward compatible with existing behavior
KIP-262: Metadata should include number of state stores for taskRejected since metadata is no longer required.
KIP-263: Allow broker to skip sanity check of inactive segments on broker startupWithdrawn because the solution we agreed on does not require interface change
KIP-286: producer.send() should not block on metadata updateWithdrawn because the benefit of not having to wait for metadata is probably not worth the complexity added in producer.
KIP-288: [DISCARDED] Consumer.poll() timeout semantic change and new waitForAssignment methodDiscarded in deference to KIP-266
KIP-310: Add a Kafka Source Connector to Kafka ConnectWithdrawn in favor of KIP-382.
KIP-327: Add describe all topics API to AdminClientWithdrawn in-favor of filter support in Metadata API and KIP-142
KIP-344: The auto-generated client id should be passed to MetricsReporterBug fix so KIP not needed
KIP-364: Remove implicit Materialized, Consumed and ProducedSubsumed by KIP-365
KIP-404: Add Kafka Connect configuration parameter for disabling WADL output on OPTIONS request

Discarded, reported as a bug by

serverASF JIRA