Status

Current state: "Accepted"

Discussion thread: https://lists.apache.org/thread/8oqzzko6rhmz08g8s6z1f2p9j1cvrn6v

Vote thread: https://lists.apache.org/thread/0ph0hwcfqdsfx3q3tbwow419cwj1glc1

JIRA: KAFKA-19237 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The Kafka Remote Log Manager currently uses three different thread pools to handle various types of tasks, but with inconsistent configuration naming and behavior, leading to several issues:

  • Naming Inconsistency: The two newer thread pool configurations follow the pattern remote.log.manager.[task-type].thread.pool.size, but the original configuration remote.log.manager.thread.pool.size doesn't follow this pattern, despite now being specifically used for follower partition tasks.

  • Inconsistent Dynamic Configuration Support: remote.log.manager.copier.thread.pool.size and remote.log.manager.expiration.thread.pool.size support dynamic configuration adjustments, while remote.log.manager.thread.pool.size does not, limiting system flexibility during runtime.

  • Potential Confusion: The configuration name remote.log.manager.thread.pool.size suggests a general-purpose thread pool, but it now only handles follower partition tasks, which may mislead users about its actual purpose.

Although the original plan was to deprecate remote.log.manager.thread.pool.size after the implementation of KIP-950, it was retained for follower tasks, and the intended refactoring was not fully carried out.

To resolve these issues, this KIP propose introducing a new dynamic configuration, remote.log.manager.follower.thread.pool.size, while marking the old configuration as deprecated. This change would enhance consistency and maintainability, providing users with a more intuitive and flexible configuration experience.

Public Interfaces

  • Introduce a new broker config: remote.log.manager.follower.thread.pool.size

    Configuration: remote.log.manager.follower.thread.pool.size
Description: Size of the thread pool used in scheduling follower tasks to read the highest-uploaded remote-offset for follower partitions.
Type: Int
Default: 2
Valid values: [1,...]

  • Mark remote.log.manager.thread.pool.size as deprecated

Proposed Changes

The proposal introduces a new configuration for the Remote Log Manager: remote.log.manager.follower.thread.pool.size.

This configuration will support dynamic changes, allowing thread counts to be adjusted smoothly (with step increments from x/2 to x*2).

Compatibility, Deprecation, and Migration Plan

To ensure backward compatibility, the deprecated configuration remote.log.manager.thread.pool.size will still be functional. However, it should be removed in version 5.0.

The logic for RemoteLogManagerConfig#remoteLogManagerFollowerThreadPoolSize is shown below.

public int remoteLogManagerFollowerThreadPoolSize() {
    if (config.originals().containsKey(REMOTE_LOG_MANAGER_FOLLOWER_THREAD_POOL_SIZE_PROP)) {
        return config.getInt(REMOTE_LOG_MANAGER_FOLLOWER_THREAD_POOL_SIZE_PROP);
    } else {
        return config.getInt(REMOTE_LOG_MANAGER_THREAD_POOL_SIZE_PROP);
    }
}

This method first checks whether the user has explicitly set remote.log.manager.follower.thread.pool.size. If not, it falls back to remote.log.manager.thread.pool.size.

So remote.log.manager.follower.thread.pool.size always takes precedence, regardless of whether it's configured statically in the broker config file or dynamically at runtime.

For example, if the user sets remote.log.manager.thread.pool.size=10 and later dynamically configures remote.log.manager.follower.thread.pool.size=8, the follower thread pool size will be 8.

Test Plan

The patch will include both unit and integration tests to ensure full coverage.

Rejected Alternatives

  • Reusing the existing remote.log.manager.thread.pool.size and making it dynamic was considered. However, as outlined in the motivation section, the current config name is misleading and inconsistent with other thread pool size configurations.
  • No labels