Status

Current state: Accepted

Discussion threads: here (the original), here (a new one as PonyMail didn't manage to associate the new messages with the old one)

JIRA: KAFKA-16445 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Connect REST API provides an endpoint /connectors/<id>/config for a connector's configuration. The endpoint allows to GET a connector's configuration and to PUT a new configuration. The only way to update an existing connector's configuration is to get, modify, and put it using this endpoint. This may cause two issues:

  1. Potential lost modifications due to non-atomicity of the update. The configuration can be changed between GET and PUT in an unrelated way. These changes will be overwritten.
  2. Inconvenience, especially when working in a command line using tools like cURL.

HTTP has PATCH method specifically designed for partial updates. It can solve or mitigate the listed issues.

Public Interfaces

We will add PATCH method to /connectors/<id>/config endpoint in Connect REST API.

As JSON null values aren't allowed in the connector configs, null will serve as a tombstore value used for deleting existing fields from configs.

Responses follow the model of the configuration PUT endpoint. If the patch was successfully applied, the response code is 200 and the body is a JSON object with the updated connector information (name, type, config, and tasks), for example:

{
    "name": "my-connector",
    "config":
    {
        "name": "my-connector",
        "sample_config_2": "test_config_2",
        "sample_config": "test_config_new"
    },
    "tasks":
    [
        {
            "connector": "my-connector",
            "task": 0
        },
        {
            "connector": "my-connector",
            "task": 1
        }
    ],
    "type": "sink"
}

In case of errors, the response code matches the error type (e.g. 400 in case of a config validation error; 404 if the connector is not found; 500 in case of other server-side errors) and the body is a JSON object with the error details:


{
    "error_code": 400,
    "message": "Connector configuration is invalid and contains the following 1 error(s):\n...\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`"
}

Proposed Changes

We will add PATCH method definition into ConnectorsResource, which will call the actual logic in Herder and its child classes StandaloneHerder and DistributedHerder. Due to the way how the herders are implemented now, it's difficult to guarantee 100% race-free partial updates. Besides, other similar race scenarios exist, so we keep out of scope of this KIP. The implementation will attempt to keep the race window as narrow as possible.

There is a draft pull request: https://github.com/apache/kafka/pull/6934

Compatibility, Deprecation, and Migration Plan

This change is backward compatible.

Rejected Alternatives

No rejected alternatives.


  • No labels