Status

Current state: Accepted

Voting thread: https://lists.apache.org/thread/53g4g2jqh7bs0g221l3obb3mjcon84q2

Discussion thread: https://lists.apache.org/thread/5gqlj8x3pxr816j1bcq39q9qbss2d0xk

Co-Author: ujjwal kalia (Ujjwal Kalia)

Motivation

Apache Kafka added support for the OAuth 2.0 client_credentials grant type in KIP-768. The current implementation uses the traditional client authentication method where a client authenticates using a client ID and client secret passed via HTTP Basic authentication. While functional, this approach has several limitations in modern cloud-native and security-conscious environments.

This KIP proposes adding support for client assertion as an alternative authentication method for the client_credentials grant type, as defined in RFC 7521 and RFC 7523. This enhancement addresses three key motivators:

Enhanced Security

The current client secret approach requires storing long-lived secrets in plain text within configuration files. This creates several security risks:

Secrets can be accidentally committed to version control
Configuration files may be inadvertently exposed through backups, logs, or monitoring systems
Rotating secrets requires coordinating updates across all clients simultaneously
Compromised secrets provide long-term access until manually rotated

Client assertion authentication eliminates these risks by using cryptographic signatures instead of plain text secrets:

Short-lived assertions: Each assertion is valid only for a brief period (typically 5-10 minutes), limiting the window of exposure
Private keys never leave the client: Only the signed assertion is transmitted, not the key material itself
Cryptographic proof: The assertion provides cryptographic proof of the client's identity without revealing the secret
Easier rotation: Private keys can be rotated independently with automatic file reloading

Provider Requirements

Some OAuth 2.0 identity providers require or strongly prefer client assertion over client secrets for security and compliance reasons. Organizations using these providers cannot currently use Kafka's OAuth support with the client_credentials grant type. Supporting client assertion makes Kafka compatible with any RFC 7523-compliant identity provider.

Industry Standard

Client assertion authentication is a widely-adopted OAuth 2.0 best practice, particularly in enterprise and regulated environments. It is the recommended authentication method in many security frameworks and compliance standards. Supporting this standard ensures Kafka follows industry best practices for OAuth authentication.

Background: Grant Types vs Client Authentication Methods

To avoid confusion, it's important to clarify the distinction between OAuth grant types and client authentication methods, as both can involve JWTs:

Grant Type

The grant type defines what is being requested from the OAuth provider. It appears as the grant_type parameter in the token request:

client_credentials: The client requests an access token using its own credentials (this KIP)
urn:ietf:params:oauth:grant-type:jwt-bearer: The client requests an access token by exchanging a JWT assertion as the authorization grant itself (KIP-1139)

Client Authentication Method

The client authentication method defines how the client proves its identity to the OAuth provider when making the token request. RFC 6749 Section 2.3 defines multiple authentication methods:

Client Secret (RFC 6749 Section 2.3.1): Traditional username/password-style authentication using client_id and client_secret sent via HTTP Basic authentication
Client Assertion (RFC 7523 Section 2.2): JWT-based proof sent as client_assertion and client_assertion_type parameters

Relationship to KIP-1139

KIP-1139 added support for the jwt-bearer grant type, where a JWT assertion serves as the authorization grant itself. The identity provider exchanges the assertion for an access token.

This KIP adds client assertion authentication to the existing client_credentials grant type. Here, the JWT assertion proves the client's identity (authentication), while the grant type remains client_credentials.

Both features use similar JWT assertion infrastructure, so this KIP builds on KIP-1139 by reusing its assertion creation, signing, and file caching components for a different use case.

Public Interfaces

No new public interfaces are introduced by this KIP.

All changes are internal implementation classes within the org.apache.kafka.common.security.oauthbearer.internals.secured package:

ClientAssertionRequestFormatter (new internal class)
ClientSecretRequestFormatter (renamed from ClientCredentialsRequestFormatter, internal class)
ClientCredentialsRequestFormatterFactory (new internal class)

No new configuration properties are added. All assertion-related configurations were introduced in KIP-1139.

Existing client code requires no changes. The authentication method is automatically detected based on which configurations are present.

Proposed Changes

This KIP enhances the existing client_credentials grant type support to automatically use client assertion authentication when assertion-related configurations are present, while maintaining full backward compatibility with existing client secret configurations.

Core Implementation Changes

New ClientAssertionRequestFormatter Class

A new ClientAssertionRequestFormatter class (package org.apache.kafka.common.security.oauthbearer.internals.secured) implements the HttpRequestFormatter interface to format OAuth token requests using client assertion authentication.

The formatter creates HTTP requests with:

Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer
client_assertion=<signed JWT>
Optional client_id parameter (if configured)
Optional scope parameter (if configured)

The class accepts a CloseableSupplier<String> that provides the JWT assertion on demand, allowing assertions to be regenerated with fresh timestamps for each token request.

public interface CloseableSupplier<T> extends Supplier<T>, Closeable { }

Internal Class Rename

To improve code clarity and distinguish between the two authentication methods, the existing ClientCredentialsRequestFormatter class is renamed to ClientSecretRequestFormatter. This is purely an internal implementation change with no impact on public APIs.

Enhanced ClientCredentialsRequestFormatterFactory with Three-Tier Fallback

The ClientCredentialsRequestFormatterFactory is enhanced with an intelligent three-tier fallback mechanism for determining which authentication method to use for the client_credentials grant:

First Preference - Pre-generated Assertion File: If sasl.oauthbearer.assertion.file is configured, use ClientAssertionRequestFormatter with the assertion read from the specified file. This is useful for testing or when assertions are managed by external tools.

Second Preference - Locally-Generated Assertion: If the assertion file is not configured but sasl.oauthbearer.assertion.claim.iss is configured, use ClientAssertionRequestFormatter with a locally-generated, self-signed JWT assertion created using the configured private key.

Third Preference - Client Secret (Fallback): If no assertion-related configurations are present, use ClientSecretRequestFormatter with traditional client ID and client secret authentication. This maintains full backward compatibility with existing configurations.

The factory detects the presence of any assertion-related configuration (sasl.oauthbearer.assertion.claim.iss or sasl.oauthbearer.assertion.file) to determine whether client assertion authentication should be used. If either is present, it delegates to AssertionSupplierFactory to create the appropriate assertion supplier, which then handles the sub-choice between file-based and locally-generated assertions.

HttpRequestFormatter Decision Diagram

Runtime Behavior: No Security Downgrade on Failure

The three-tier mechanism is a configuration-time decision only, not a runtime fallback. Once the authentication method is selected during initialization, it is used for the lifetime of the client. If client assertion authentication is configured but fails at runtime (e.g., signing error, private key unreadable, IdP rejects assertion), the error will propagate and the client will not silently fall back to client secret authentication. This is a deliberate design decision to prevent unintended security downgrades. Users who configure client assertion have made an explicit security choice, and the system respects that choice even in failure scenarios.

Reused Infrastructure from KIP-1139

This KIP leverages the assertion infrastructure introduced in KIP-1139, requiring no new configuration properties:

Component Architecture

AssertionSupplierFactory: Factory for creating assertion suppliers based on configuration
AssertionCreator interface: Defines the contract for creating signed JWT assertions
- DefaultAssertionCreator: Creates assertions dynamically using a private key
- FileAssertionCreator: Reads pre-generated assertions from a file
File caching with automatic reload: Private keys and assertion files are cached in memory but automatically reloaded when the underlying files change, supporting secret rotation without client restarts
All assertion-related configurations (no new configs required):
- sasl.oauthbearer.assertion.algorithm: Signing algorithm (RS256 or ES256)
- sasl.oauthbearer.assertion.private.key.file: Path to private key file for signing
- sasl.oauthbearer.assertion.private.key.passphrase: Optional passphrase for encrypted private keys
- sasl.oauthbearer.assertion.file: Path to pre-generated assertion file
- sasl.oauthbearer.assertion.template.file: Path to JSON template file for JWT headers/claims
- sasl.oauthbearer.assertion.claim.iss: Issuer claim for the assertion
- sasl.oauthbearer.assertion.claim.sub: Subject claim for the assertion
- sasl.oauthbearer.assertion.claim.aud: Audience claim for the assertion
- sasl.oauthbearer.assertion.claim.exp.seconds: Expiration time in seconds (default: 300)
- sasl.oauthbearer.assertion.claim.nbf.seconds: Not-before time in seconds (default: 60)
- sasl.oauthbearer.assertion.claim.jti.include: Whether to include a unique JWT ID claim (default: false)

OAuth Client Assertion Flow

When client assertion authentication is used, the following flow occurs:

End-to-End Flow Diagram

Configuration Loading: The Kafka client reads the OAuth configuration, including assertion-related settings (private key file, claim values, etc.)
Assertion Creation: The client creates a signed JWT assertion:
- If using locally-generated assertions: Creates a JWT with configured claims (iss, sub, aud, exp, nbf, optional jti) and signs it using the configured private key and algorithm (RS256 or ES256)
- If using file-based assertions: Reads the pre-generated JWT from the specified file
- Token Request: The client sends an HTTP POST to the token endpoint with:
```
POST /oauth/token HTTP/1.1
Host: identity-provider.com
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer
&client_assertion=<signed-jwt>
&scope=kafka.read+kafka.write
```
Assertion Validation: The identity provider validates the assertion:
- Verifies the JWT signature using the client's registered public key
- Checks the claims (issuer, audience, expiration, etc.)
- Confirms the client is authorized for the requested scope

Token Response: If the assertion is valid, the identity provider returns an access token:

{
  "access_token": "eyJhbGciOiJSUzI1NiIs...",
  "token_type": "Bearer",
  "expires_in": 3600
}

Broker Authentication: The client uses the access token to authenticate with the Kafka broker using the existing SASL/OAUTHBEARER mechanism

Configuration Examples

Example 1: Client Assertion with Dynamically Generated JWT

This is the recommended configuration for production use (second preference in fallback order):

# OAuth token endpoint
sasl.oauthbearer.token.endpoint.url=https://identity-provider.com/oauth/token

# Private key for signing assertions
sasl.oauthbearer.assertion.private.key.file=/path/to/private-key.pem
sasl.oauthbearer.assertion.algorithm=RS256

# Assertion claims
sasl.oauthbearer.assertion.claim.iss=my-kafka-client
sasl.oauthbearer.assertion.claim.sub=my-service-account
sasl.oauthbearer.assertion.claim.aud=https://identity-provider.com
sasl.oauthbearer.assertion.claim.exp.seconds=300

# Optional scope
sasl.oauthbearer.scope=kafka.read kafka.write

# JAAS configuration
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

System Property Requirements:

-DALLOWED_SASL_OAUTHBEARER_URLS_CONFIG=https://identity-provider.com/oauth/token
-DALLOWED_SASL_OAUTHBEARER_FILES_CONFIG=/path/to/private-key.pem

Example 2: Client Assertion with Pre-generated JWT File

This configuration is useful for testing or when assertions are managed externally (first preference in fallback order):

# OAuth token endpoint
sasl.oauthbearer.token.endpoint.url=https://identity-provider.com/oauth/token

# Pre-generated assertion file
sasl.oauthbearer.assertion.file=/path/to/assertion.jwt

# Optional scope
sasl.oauthbearer.scope=kafka.read

# JAAS configuration
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

System Property Requirements:

-DALLOWED_SASL_OAUTHBEARER_URLS_CONFIG=https://identity-provider.com/oauth/token
-DALLOWED_SASL_OAUTHBEARER_FILES_CONFIG=/path/to/assertion.jwt

Example 3: Traditional Client Secret (Existing Behavior)

This existing configuration continues to work unchanged (third preference/fallback in fallback order):

# OAuth token endpoint
sasl.oauthbearer.token.endpoint.url=https://identity-provider.com/oauth/token

# Client credentials
sasl.oauthbearer.client.credentials.client.id=my-client-id
sasl.oauthbearer.client.credentials.client.secret=my-secret

# Optional scope
sasl.oauthbearer.scope=kafka.read

# JAAS configuration
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

System Property Requirements:

-DALLOWED_SASL_OAUTHBEARER_URLS_CONFIG=https://identity-provider.com/oauth/token

Example 4: Client Assertion with Encrypted Private Key

For enhanced security, the private key can be encrypted with a passphrase:

sasl.oauthbearer.token.endpoint.url=https://identity-provider.com/oauth/token 
sasl.oauthbearer.assertion.private.key.file=/path/to/encrypted-private-key.pem
sasl.oauthbearer.assertion.private.key.passphrase=my-secure-passphrase
sasl.oauthbearer.assertion.algorithm=RS256
sasl.oauthbearer.assertion.claim.iss=my-kafka-client
sasl.oauthbearer.assertion.claim.sub=my-service-account
sasl.oauthbearer.assertion.claim.aud=https://identity-provider.com
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

Compatibility, Deprecation, and Migration Plan

Full Backward Compatibility

This KIP maintains complete backward compatibility with existing OAuth configurations:

Existing client_credentials configurations work unchanged: Any configuration using client ID and client secret continues to function exactly as before
Automatic detection: The three-tier fallback mechanism automatically selects the appropriate authentication method based on which configurations are present
No migration required: Existing users can continue using client secret authentication indefinitely
No breaking changes: No public APIs, configuration properties, or behaviors are deprecated or removed
JAAS Compatibility: Compatible with top level jaas configs or JAAS options

Internal Class Rename

The ClientCredentialsRequestFormatter class is renamed to ClientSecretRequestFormatter. This is purely an internal implementation change:

The class resides in the org.apache.kafka.common.security.oauthbearer.internals.secured package (internal)
No public APIs reference this class
No impact on existing deployments or configurations

Optional Migration Path

Users who wish to migrate from client secrets to client assertions can do so seamlessly:

Add assertion configurations: Simply add the assertion-related configurations (private key file, claims, etc.) to the client configuration
Automatic detection: The ClientCredentialsRequestFormatterFactory will automatically detect the assertion configurations and switch to client assertion authentication
Remove client secret configurations: Once verified, the old client.id and client.secret configurations can be removed (optional)
Rollback capability: Removing the assertion configurations will cause the client to fall back to client secret authentication

The three-tier preference system allows for gradual migration and testing:

Test with pre-generated assertion files first
Migrate to locally-generated assertions once comfortable
Keep client secret as a fallback during the transition period

Test Plan

The following tests will be implemented to ensure the correctness and backward compatibility of this feature:

Unit Tests for ClientAssertionRequestFormatter
Unit Tests for ClientCredentialsRequestFormatterFactory
Unit Tests for AssertionSupplierFactory Integration

Integration Tests

End-to-End Flow with KeyCloakContainer:
- Set up a KeyCloak testcontainer with client assertions enabled.
- Verify that Kafka clients can successfully obtain tokens using client assertions
- Verify the actual HTTP requests sent to the token endpoint match RFC 7523 specifications
Token Retrieval and Usage: Verify that access tokens obtained via client assertion can be used to authenticate with Kafka brokers
Test token refresh flows with connections.max.reauth.ms (Simulation only)
File-Based Assertions: Test end-to-end flow using pre-generated assertion files
Locally-Generated Assertions: Test end-to-end flow using dynamically generated assertions

Compatibility Tests

Existing Client Secret Flow: Verify that existing configurations using client ID and client secret continue to work without any changes
Migration Scenarios:
- Test adding assertion configs to an existing client secret configuration
- Test removing assertion configs to fall back to client secret
- Test switching between file-based and locally-generated assertions

Error Handling Tests

Invalid Assertions: Verify appropriate errors when assertions are malformed, expired, or have invalid signatures
Missing Configurations: Verify clear error messages when required configurations are missing
Network Failures: Verify appropriate retry behavior and error handling when token endpoint is unavailable

Rejected Alternatives

Adding a New Configuration to Select Authentication Method

We considered adding an explicit configuration property (e.g., sasl.oauthbearer.client.authentication.method) to let users choose between "client_secret" and "client_assertion".

Rejected because:

Auto-detection based on configuration presence is simpler and more intuitive
Avoids adding yet another configuration property for users to understand
Natural inference from user intent: if assertion configs are present, use assertions
The three-tier fallback provides flexibility without requiring explicit mode selection
Reduces configuration complexity and potential for misconfiguration

Creating a Separate Grant Type

We considered treating client assertion as a completely separate grant type with its own JwtRetriever implementation.

Rejected because:

Client assertion is fundamentally an authentication method for client_credentials, not a separate grant type
RFC 7523 Section 2.2 explicitly defines this as "Using JWTs for Client Authentication", not as a grant type
Would create confusion about OAuth terminology and lead users to misunderstand the distinction
Would require duplicate code and configuration for what is essentially the same grant type with a different authentication method
The grant type is still client_credentials; only the authentication method changes

Requiring Migration of Existing Configurations

We considered requiring users to explicitly migrate their configurations to a new format or explicitly opt into assertion support.

Rejected because:

Breaks backward compatibility unnecessarily
Creates disruption for existing Kafka deployments
The seamless three-tier fallback ensures existing deployments continue working without any changes
Users can adopt client assertion at their own pace without forced migration
No security or functional reason to force migration

Using a Single Fallback Instead of Three-Tier

We considered a simpler two-tier fallback (assertion vs. client secret) without the distinction between file-based and locally-generated assertions.

Rejected because:

The file-based vs. locally-generated distinction is already done in KIP-1139
Maintaining consistency with KIP-1139 is important for code reuse
Different use cases benefit from different approaches (testing, external tools, production)
The three-tier approach provides maximum flexibility without added complexity for users

References

RFC 6749: OAuth 2.0 Authorization Framework RFC 6749: The OAuth 2.0 Authorization Framework - Foundation of OAuth 2.0
RFC 7521: Assertion Framework for OAuth 2.0 Client Authentication and Authorization Grants RFC 7521: Assertion Framework for OAuth 2.0 Client Authentication and Authorization Grants - Framework for using assertions in OAuth
RFC 7523: JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants RFC 7523: JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants - Specific JWT profile for OAuth assertions
- Section 2.2: Using JWTs for Client Authentication - The specific section this KIP implements
KIP-1139: Add support for OAuth jwt-bearer grant KIP-1139: Add support for OAuth jwt-bearer grant type - Apache Kafka - Apache Software Foundation - Introduced assertion infrastructure that this KIP reuses
KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC - Apache Kafka - Apache Software Foundation - Original OAuth support in Kafka

Space shortcuts

Child pages

Status

Motivation

Enhanced Security

Provider Requirements

Industry Standard

Background: Grant Types vs Client Authentication Methods

Grant Type

Client Authentication Method

Relationship to KIP-1139

Public Interfaces

Proposed Changes

Core Implementation Changes

New ClientAssertionRequestFormatter Class

Internal Class Rename

Enhanced ClientCredentialsRequestFormatterFactory with Three-Tier Fallback

Reused Infrastructure from KIP-1139

OAuth Client Assertion Flow

Configuration Examples

Example 1: Client Assertion with Dynamically Generated JWT

Example 2: Client Assertion with Pre-generated JWT File

Example 3: Traditional Client Secret (Existing Behavior)

Example 4: Client Assertion with Encrypted Private Key

Compatibility, Deprecation, and Migration Plan

Full Backward Compatibility

Internal Class Rename

Optional Migration Path

Test Plan

Integration Tests

Compatibility Tests

Error Handling Tests

Rejected Alternatives

Adding a New Configuration to Select Authentication Method

Creating a Separate Grant Type

Requiring Migration of Existing Configurations

Using a Single Fallback Instead of Three-Tier

References

Space shortcuts

Child pages

KIP-1258: Add Support for OAuth Client Assertion to client_credentials Grant Type

Status

Motivation

Enhanced Security

Provider Requirements

Industry Standard

Background: Grant Types vs Client Authentication Methods

Grant Type

Client Authentication Method

Relationship to KIP-1139

Public Interfaces

Proposed Changes

Core Implementation Changes

New ClientAssertionRequestFormatter Class

Internal Class Rename

Enhanced ClientCredentialsRequestFormatterFactory with Three-Tier Fallback

Reused Infrastructure from KIP-1139

OAuth Client Assertion Flow

Configuration Examples

Example 1: Client Assertion with Dynamically Generated JWT

Example 2: Client Assertion with Pre-generated JWT File

Example 3: Traditional Client Secret (Existing Behavior)

Example 4: Client Assertion with Encrypted Private Key

Compatibility, Deprecation, and Migration Plan

Full Backward Compatibility

Internal Class Rename

Optional Migration Path

Test Plan

Integration Tests

Compatibility Tests

Error Handling Tests

Rejected Alternatives

Adding a New Configuration to Select Authentication Method

Creating a Separate Grant Type

Requiring Migration of Existing Configurations

Using a Single Fallback Instead of Three-Tier

References