Dead Letter Channel
It is common for a temporary outage or database deadlock to cause a message to fail to process; but the chances are if its tried a few more times with some time delay then it will complete fine. So we typically wish to use some kind of redelivery policy to decide how many times to try redeliver a message and how long to wait before redelivery attempts.
The RedeliveryPolicy defines how the message is to be redelivered. You can customize things like
- how many times a message is attempted to be redelivered before it is considered a failure and sent to the dead letter channel
- the initial redelivery timeout
- whether or not exponential backoff is used (i.e. the time between retries increases using a backoff multiplier)
- whether to use collision avoidance to add some randomness to the timings
- delay pattern (see below for details)
- Camel 2.11: whether to allow redelivery during stopping/shutdown
Once all attempts at redelivering the message fails then the message is forwarded to the dead letter queue.
About moving Exchange to dead letter queue and using handled
Handled on Dead Letter Channel
When all attempts of redelivery have failed the Exchange is moved to the dead letter queue (the dead letter endpoint). The exchange is then complete and from the client point of view it was processed. As such the Dead Letter Channel have handled the Exchange.
For instance configuring the dead letter channel as:
Using the Fluent Builders
Using the Spring XML Extensions
The Dead Letter Channel above will clear the caused exception (
setException(null)), by moving the caused exception to a property on the Exchange, with the key
Exchange.EXCEPTION_CAUGHT. Then the Exchange is moved to the
"jms:queue:dead" destination and the client will not notice the failure.
About moving Exchange to dead letter queue and using the original message
The option useOriginalMessage is used for routing the original input message instead of the current message that potentially is modified during routing.
For instance if you have this route:
The route listen for JMS messages and validates, transforms and handle it. During this the Exchange payload is transformed/modified. So in case something goes wrong and we want to move the message to another JMS destination, then we can configure our Dead Letter Channel with the useOriginalMessage option. But when we move the Exchange to this destination we do not know in which state the message is in. Did the error happen in before the transformOrder or after? So to be sure we want to move the original input message we received from
jms:queue:order:input. So we can do this by enabling the useOriginalMessage option as shown below:
Then the messages routed to the
jms:queue:dead is the original input. If we want to manually retry we can move the JMS message from the failed to the input queue, with no problem as the message is the same as the original we received.
When Dead Letter Channel is doing redeliver its possible to configure a Processor that is executed just before every redelivery attempt. This can be used for the situations where you need to alter the message before its redelivered. See below for sample.
Redelivery default values
Redelivery is disabled by default.
The default redeliver policy will use the following values:
- redeliverDelay=1000L (1 second)
- maximumRedeliveryDelay = 60 * 1000L (60 seconds)
- And the exponential backoff and collision avoidance is turned off.
- The retriesExhaustedLogLevel are set to LoggingLevel.ERROR
- The retryAttemptedLogLevel are set to LoggingLevel.DEBUG
- Stack traces is logged for exhausted messages from Camel 2.2 onwards.
- Handled exceptions is not logged from Camel 2.3 onwards
- logExhaustedMessageHistory is true for default error handler, and false for dead letter channel.
The maximum redeliver delay ensures that a delay is never longer than the value, default 1 minute. This can happen if you turn on the exponential backoff.
The maximum redeliveries is the number of re delivery attempts. By default Camel will try to process the exchange 1 + 5 times. 1 time for the normal attempt and then 5 attempts as redeliveries.
Setting the maximumRedeliveries to a negative value such as -1 will then always redelivery (unlimited).
Setting the maximumRedeliveries to 0 will disable any re delivery attempt.
Camel will log delivery failures at the DEBUG logging level by default. You can change this by specifying retriesExhaustedLogLevel and/or retryAttemptedLogLevel. See ExceptionBuilderWithRetryLoggingLevelSetTest for an example.
You can turn logging of stack traces on/off. If turned off Camel will still log the redelivery attempt. Its just much less verbose.
Redeliver Delay Pattern
Delay pattern is used as a single option to set a range pattern for delays. If used then the following options does not apply: (delay, backOffMultiplier, useExponentialBackOff, useCollisionAvoidance, maximumRedeliveryDelay).
The idea is to set groups of ranges using the following syntax:
limit:delay;limit 2:delay 2;limit 3:delay 3;...;limit N:delay N
Each group has two values separated with colon
- limit = upper limit
- delay = delay in millis
And the groups is again separated with semi colon.
The rule of thumb is that the next groups should have a higher limit than the previous group.
Lets clarify this with an example:
That gives us 3 groups:
Resulting in these delays for redelivery attempt:
- Redelivery attempt number 1..4 = 0 millis (as the first group start with 5)
- Redelivery attempt number 5..9 = 1000 millis (the first group)
- Redelivery attempt number 10..19 = 5000 millis (the second group)
- Redelivery attempt number 20.. = 20000 millis (the last group)
Note: The first redelivery attempt is 1, so the first group should start with 1 or higher.
You can start a group with limit 1 to eg have a starting delay:
- Redelivery attempt number 1..4 = 1000 millis (the first group)
- Redelivery attempt number 5.. = 5000 millis (the last group)
There is no requirement that the next delay should be higher than the previous. You can use any delay value you like. For example with
delayPattern=1:5000;3:1000 we start with 5 sec delay and then later reduce that to 1 second.
When a message is redelivered the DeadLetterChannel will append a customizable header to the message to indicate how many times its been redelivered.
Before Camel 2.6: The header is CamelRedeliveryCounter, which is also defined on the
Starting with 2.6: The header CamelRedeliveryMaxCounter, which is also defined on the
Exchange.REDELIVERY_MAX_COUNTER, contains the maximum redelivery setting. This header is absent if you use
retryWhile or have unlimited maximum redelivery configured.
And a boolean flag whether it is being redelivered or not (first attempt)
The header CamelRedelivered contains a boolean if the message is redelivered or not, which is also defined on the
Dynamically calculated delay from the exchange
In Camel 2.9 and 2.8.2: The header is CamelRedeliveryDelay, which is also defined on the
Is this header is absent, normal redelivery rules apply.
Which endpoint failed
Available as of Camel 2.1
Exchange.TO_ENDPOINT have the constant value
This information is updated when Camel sends a message to any endpoint. So if it exists its the last endpoint which Camel send the Exchange to.
When for example processing the Exchange at a given Endpoint and the message is to be moved into the dead letter queue, then Camel also decorates the Exchange with another property that contains that last endpoint:
Exchange.FAILURE_ENDPOINT have the constant value
This allows for example you to fetch this information in your dead letter queue and use that for error reporting.
This is useable if the Camel route is a bit dynamic such as the dynamic Recipient List so you know which endpoints failed.
Notice: These information is kept on the Exchange even if the message was successfully processed by a given endpoint, and then later fails for example in a local Bean processing instead. So beware that this is a hint that helps pinpoint errors.
Now suppose the route above and a failure happens in the
foo bean. Then the
Exchange.FAILURE_ENDPOINT will still contain the value of
Available as of Camel 2.16
Before the exchange is sent to the dead letter queue, you can use onPrepare to allow a custom
Processor to prepare the exchange, such as adding information why the Exchange failed. For example the following processor adds a header with the exception message
Then configure the error handler to use the processor as follows:
Configuring this from XML DSL is as shown:
The onPrepare is also available using the default error handler.
Which route failed
Available as of Camel 2.10.4/2.11
Exchange.FAILURE_ROUTE_ID have the constant value
This allows for example you to fetch this information in your dead letter queue and use that for error reporting.
Control if redelivery is allowed during stopping/shutdown
Available as of Camel 2.11
Prior to Camel 2.10, Camel will perform redelivery while stopping a route, or shutting down Camel. This has improved a bit in Camel 2.10 onwards, as Camel will not perform redelivery attempts when shutting down aggressively (eg during Graceful Shutdown and timeout hit). From Camel 2.11 onwards there is a new option
allowRedeliveryWhileStopping which you can use to control if redelivery is allowed or not; notice that any in progress redelivery will still be executed. This option can only disallow any redelivery to be executed after the stopping of a route/shutdown of Camel has been triggered. If a redelivery is dissallowed then a
RejectedExcutionException is set on the Exchange and the processing of the Exchange stops. This means any consumer will see the Exchange as failed due the
The default value is
true to be backwards compatible as before. For example the following sample shows how to do this with Java DSL and XML DSL
And the sample sample with XML DSL
The following example shows how to configure the Dead Letter Channel configuration using the DSL
You can also configure the RedeliveryPolicy as this example shows
How can I modify the Exchange before redelivery?
When Dead Letter Channel is doing redeliver its possible to configure a Processor that is executed just before every redelivery attempt. This can be used for the situations where you need to alter the message before its redelivered.
Here we configure the Dead Letter Channel to use our processor
MyRedeliveryProcessor to be executed before each redelivery.
And this is the processor
MyRedeliveryProcessor where we alter the message.
How can I log what caused the Dead Letter Channel to be invoked?
You often need to know what went wrong that caused the Dead Letter Channel to be used and it does not offer logging for this purpose. So the Dead Letter Channel's endpoint can be set to a endpoint of our own (such as
direct:deadLetterChannel). We write a route to accept this Exchange and log the Exception, then forward on to where we want the failed Exchange moved to (which might be a DLQ queue for instance). See also http://stackoverflow.com/questions/13711462/logging-camel-exceptions-and-sending-to-the-dead-letter-channel
Using This Pattern
If you would like to use this EIP Pattern then please read the Getting Started, you may also find the Architecture useful particularly the description of Endpoint and URIs. Then you could try out some of the Examples first before trying this pattern out.