This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-349: Priorities for Source Topics

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

We use Kafka to process the asynchronous events of our Document Management System such as preview generation, indexing for search etc.
The traffic gets generated via Web and Desktop Sync application. In such cases, we had to prioritize the traffic from web and consume them first.  But this might lead to the starvation of events from sync if the consumer speed is slow and the event rate is high from web.  A solution to handle the starvation with a timeout after which the events are consumed normally for a specified period of time would be great and help us use our resources effectively.

Public Interfaces

The addition of new subscribe API that allows caller to prioritize topics.  New class TopicPriority constructor

...

Note that the issue of starvation of lower priority topics has been discussed in the most recent thread.  The above API will be extended to handle starvation.  One idea is to check the delta between the oldest messages in the different topics.  The consumer would then revert to using no priorities when the delta reaches a max threshold for the delta, and then re-enable prioritization after achieving a min threshold. and is intended by design. 

Compatibility, Deprecation, and Migration Plan

...