DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
The records are written onto a DLQ topic by a broker which is intrinsically capable of writing to all topics. However, there is a security angle to consider here. If an application writes a record onto a topic T1 for which it has permission, and then causes the record to be treated as undeliverable by a share group with a DLQ topic configured, perhaps by rejecting it explicitly or hitting the maximum number of delivery attempts, the broker will write a DLQ record onto the DLQ topic. The application which wrote the original record onto T1 does not necessarily have permission to write onto the DLQ topic itself. Now let's say that the DLQ topic name configured is the name of a critical business topic, the original application has a convoluted way to write onto this critical topic. This is why the naming of the DLQ topic can be controlled, and why topics must be specifically configured to permit their use as DLQ topics.
In order to be used as a DLQ topic for share groups, and potentially other kinds of group in the future, the topic configuration errors.deadletterqueue.group.enable must be true . This means the administrator of the topic controls whether the topic can be used as a DLQ topic.
A broker configuration errors.deadletterqueue.topic.name.prefix contains the prefix of permitted DLQ topic names. By default, this is "dlq." meaning that DLQ topics must start with this prefix, such as "dlq.T1" . This instantly tells consuming applications that the records they see were indirectly written to this topic. The context information headers also indicate that these are potentially DLQ records.
...
Second, a DLQ topic could be considered as a cluster-wide topic. In this case, there would be a single DLQ topic on the cluster with a well-known name. It is entirely possible that the records on the central DLQ topic would have multiple schemas because the records could have originated in a variety of applications.
While it would be a good convention for DLQ topics to begin with the prefix "dlq." in all cases, some clusters implement a simple form of multi-tenancy by employing a prefix naming convention such as "emea.sales" and using prefix ACLs. In order to accommodate naming conventions such as these, the cluster administrator can choose their own DLQ topic name prefix, or even choose not to have a prefix at all thus allowing the DLQ topics to fit in with a pre-existing topic naming convention.
Public Interfaces
Configuration
...
These are dynamic configurations defined at the cluster level.
| Configuration | Description | Values |
|---|---|---|
errors.deadletterqueue.auto.create.topics.enable | Whether to create DLQ topics automatically. Note that automatic creation of the DLQ topics is independent of the auto.create.topics.enable configuration. When a DLQ topic is created automatically, the topic configuration errors.deadletterqueue.group.enable will be set to true . | Type: boolean, default: false |
errors.deadletterqueue.topic.name.prefix | The prefix of permitted dead-letter queue topic names. If this is set to "", there is no restriction on the names used for dead-letter queue topics. | Type: string, default: "dlq." |
For example, the following command would enable auto-creation of DLQ topics for the cluster.
...
Note that these configuration property names do not begin with share even though they are initially only relevant to the behavior of share groups. This is because the concept is likely to be applicable to other group types in the future.
| Configuration | Description | Values |
|---|---|---|
errors.deadletterqueue.topic.name | The name of the topic to be used as the dead-letter queue (DLQ) for undeliverable records. The topic name is blank by default, which means that there is no DLQ topic for this group. | Type: string, default: "", must not start with double underscore "_ _" to ensure that internal topics are not mistakenly used as DLQ topics |
errors.deadletterqueue.copy.record.enable | Whether to copy the record onto the dead-letter queue topic, or just to write a record containing the context information headers. | Type: boolean, default: false |
Topic configuration
| Configuration | Description | Values |
|---|---|---|
errors.deadletterqueue.group.enable | Whether the topic can be used as a DLQ topic for groups. | Type: boolean, default: false |
Records
This KIP defines a new DeliveryState of Archiving with the value of 3 which indicates a record which is in the process of being written to the dead-letter queue topic. As the details of the implementation develop, it will become clear whether this value is written to the share-group state topic, or just a state in memory like Acquired. The KIP reserves it for potential use in the existing share coordinator record schemas.
...