Status

Current state:  Under Discussion

Discussion thread: https://lists.apache.org/thread/7vkpl9k0wxdtm170xbq77o00z52h66yy

JIRA:KAFKA-20392

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The current GET /connectors endpoint in Kafka Connect supports an expand query parameter that changes the shape of the response:

• Without expand: returns a list of connector names
• With expand: returns a map of connector → expanded details

This creates a non-uniform API contract, where the same endpoint produces incompatible response schemas depending on query parameters.

This causes issues for:

• Generated clients (OpenAPI / strongly typed clients)
• API consumers expecting stable schemas
• Documentation clarity and long-term maintainability

More importantly, this violates a core REST principle:

A single endpoint should have a predictable response structure.

This KIP proposes to separate the concerns:

• Keep /connectors as a stable, simple listing endpoint
• Introduce a new endpoint for expanded connector metadata

This is the problem described in KAFKA-20392.

Goals

• One predictable response type per endpoint for the scope of this KIP (strict list vs. expanded map).
• Preserve all behavior currently available through expand (no capability loss).
• Improve clarity for clients and schema tooling.
• Provide a migration path without a breaking change in the first release.

Non-goals

• Redesigning the broader Connect REST API.
• Changing connector runtime behavior or semantics.
• Forcing every client to migrate in a single release.

Public interfaces

REST API

GET /connectors

• Response: JSON array of connector name strings only (unchanged shape when expand is absent).
• expand: Not part of the long-term contract on this URL.
  • From the release that introduces GET /connectors-details: requests that include expand still behave as today (expanded map) for one major release only; implementations MUST log a WARN on each such request with text that names GET /connectors-details as the replacement.
  • Starting the next major release: requests to GET /connectors that include expand will ignore it and will return list of connectors.
• Media type: application/json.

GET /connectors-details (new)

• Purpose: Sole supported URL for the expanded map response.
• Query parameters: Same as today: expand=status, expand=info, repeated expand for multiple values; unknown values ignored/logged as today.
• Response: Same JSON object shape as today’s  GET /connectors?expand=... (connector name → object with optional status / info entries).

Documentation / OpenAPI

• Document GET /connectors as array only in the steady state after deprecation; document the one-major-release warning window.
• Document GET /connectors-details with a single fixed response schema (object map).

• Update public Connect REST documentation and OpenAPI  to reflect deprecation of expand on GET /connectors, the warning window, and the new GET /connectors-details endpoint.

Monitoring

• Required for the deprecation window: WARN log when expand is used on GET /connectors 

Proposed Changes

New Endpoint

Introduce a new endpoint:

GET /connectors-details

This endpoint returns expanded connector metadata.

Supported query parameters:

expand=status
expand=info
expand=status&expand=info

 Response Format

Example:

{
  "my-source": {
    "status": {
      "name": "my-source",
      "connector": {
        "state": "RUNNING",
        "worker_id": "worker1:8083"
      },
      "tasks": [
        { "id": 0, "state": "RUNNING", "worker_id": "worker1:8083" }
      ],
      "type": "source"
    },
    "info": {
      "name": "my-source",
      "config": {
        "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
        "tasks.max": "1",
        "topic": "my-topic",
        "file": "/tmp/test.txt"
      },
      "tasks": [
        { "connector": "my-source", "task": 0 }
      ],
      "type": "source"
    }
  }
}

This response will be same as current response for /connectors?expand=status&expand=info

Compatibility and Migration Plan 

Release N (Introduction Phase)

Additional behavior:

• Log warning when expand is used on /connectors

Release N+1 (Deprecation Enforcement Phase)

Rationale

This phased approach:

• Preserves backward compatibility
• Provides clear migration path
• Aligns with existing Kafka API evolution practices

Test plan 

• Unit: Assert GET /connectors with no query returns List<String>; assert GET /connectors-details returns the map for status, info, and both; assert in N+1 behavior tests that GET /connectors?expand=status yields list of connectors .
• Regression: Preserve existing edge-case tests for expansion (e.g. connector not found while building the map) on the new path.

Rejected Alternatives

1. Documentation only — does not remove union-type responses on GET /connectors for clients that must support both modes today.
2. Wrapper on GET /connectors (e.g. { "names": [...], "details": {...} }) — breaks existing clients that expect a bare JSON array.
3. Content negotiation (Accept) — harder to use and operate; still overloads one resource compared to two explicit URLs.