Status

Current state: Accepted

Discussion thread:https://www.mail-archive.com/dev@kafka.apache.org/msg97405.html

JIRA:  KAFKA-8309 - Getting issue details... STATUS

Release: AK 2.3.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Connect's API provides a list method which returns all running connector names (/connectors) as well as state info methods parameterized by name (/connectors/{connectorName}/status). This leads to a situation where users of this API need to make N api calls to get information about all their running connectors.

Public Interfaces

The current `/connectors` endpoint will gain a new multi query param `?expand=(status|info)` which will return a map {connectorName->expansion->(ConnectorStateInfo|ConnectorInfo)}.

the following expansions are allowed

expandclass returned

status

ConnectorStateInfo
infoConnectorInfo


w:kafka norwood$ curl -s http://localhost:8083/connectors | jq
[
  "blah"
]




w:kafka norwood$ curl -s 'http://localhost:8083/connectors?expand=status&expand=info' | jq
{
  "blah": {
    "info": {
      "name": "blah",
      "config": {
        "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
        "file": "/tmp/lol",
        "tasks.max": "10",
        "name": "blah",
        "topic": "test-topic"
      },
      "tasks": [
        {
          "connector": "blah",
          "task": 0
        }
      ],
      "type": "source"
    },
    "status": {
      "name": "blah",
      "connector": {
        "state": "RUNNING",
        "worker_id": "10.200.25.241:8083"
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "10.200.25.241:8083"
        }
      ],
      "type": "source"
    }
  }
}


Proposed Changes

The changes are ~entirely in ConnectorsResource and the Herders. PR: https://github.com/apache/kafka/pull/6658

Compatibility, Deprecation, and Migration Plan

This change is backwards compatible.

Rejected Alternatives

  • Adding an entirely new endpoint.
    • The way that ConnectorsResource is setup right now is not conducive to adding another endpoint that is not namespaced under /connectors/{connectorName}
    • We could conceivably rework more code in ConnectorsResource to allow us to have a /connectorsExpanded endpoint and then retain the other /connectors/{connectorName}/status endpoints, but this introduces even more questions, e.g. what would be under /connectorsExpanded/{connector}/status? 
  • Add a connect rest extension.
    • This seems like overkill for basic functionality. We have all the data available in the herder now, so requiring additional install steps to get basic info seems like a bit much.
  • No labels