Table of Contents |
---|
Status
Current state: Under Discarded
Discussion thread: here
Discussion Vote thread: pending here
JIRA: pending KAFKA-6725
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
After some code analysis, we determined that there wasn't an existing API that a Connector could use to determine that the task was about to be shut down for a rebalance. After some thought we decided that this was a problem more generally about giving connectors the information about the state of the world outside the connector such that they could make intelligent decisions about how to behave in preCommit
. Giving this information to a connector gives developers writing connectors options to change their behavior if a rebalance or shutdown is pending.
We To achieve this we would like to add such an API so that the preCommit
hook of a Connector can decide if it should behave differently because the task is actively trying to shut down.
...
Proposed Changes
- We'll add the
isClosing
method to theSinkTaskContext
as mentioned above. - We'll add the implementation for that to
WorkerSinkTaskContext
- We'll add a setter named
setClosingInProgress
toWorkerSinkTaskContext
that'll change the internal state - In
WorkerSinkTask.commitOffsets
:- If the closing parameter is set to
true
, we'll invokesetClosingInProgress
on the context totrue
. This will occur before the invocation ofpreCommit
. - Before
commitOffsets
returns, it will setClosingInProgress willsetClosingInProgress
back tofalse
.
- If the closing parameter is set to
- Add a note to the
preCommit
javadoc to indicate the importance of checkingSinkTaskContext.isClosed
when running,
This change will give preCommit the opportunity to provide some additional commits that it would like Kafka Connect to make before the task is fully shut down.
...
- We considered adding a closing parameter to preCommit, but that would be a breaking change for all connectors and would require them to change their code when upgrading to the next version of Connect. It would also break binary compability.
...
- unless it was done very carefully. The additional complexity would buy us marginal extra value, if any at all, so we decided to go with the addition to the context object.