Status
Current state: Under Discussion
Discussion thread:
Vote:
JIRA:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The AlterConfigs RPC gives users a way to alter the configuration of a topic, broker, or other resource. However, the new configuration replaces any existing configuration. This makes AlterConfigs unusable in cases where the client does not know the full existing configuration before making the modification. This makes AlterConfigs less efficient, since it means that the client needs to call DescribeConfigs to retrieve the existing configuration before making any modification. It also introduces the possibility of "lost updates" when multiple clients enter a read-modify-write cycle around the same time.
Even worse, in some cases, the client may be unable to discover the existing configuration. "Sensitive" fields are not returned by DescribeConfigs. However, they will be overwritten by AlterConfigs.
In order to fix these issues, we should introduce a new RPC named incrementalAlterConfigs. The new RPC should operate incrementally, modifying only the configuration values that are specified. We should deprecate AlterConfigs.
Proposed Changes
The new IncrementalAlterConfigs API in AdminClient will take a map describing the configuration modifications, and a set describing configuration keys which should be removed.
public interface AlterConfigOp {
ConfigResource resource();
}
public class SetConfigOp extends AlterConfigOp {
public SetConfigOp(ConfigResource resource, String value);
}
public class DeleteConfigOp extends AlterConfigOp {
public DeleteConfigOp(SConfigResource resource);
}
public class AppendConfigOp extends AlterConfigOp {
public AppendConfigOp(ConfigResource resource, String value);
}
public class SubtractConfigOp extends AlterConfigOp {
public SubtractConfigOp(ConfigResource resource, String value);
}
public IncrementalAlterConfigsResult incrementalAlterConfigs(
Collection<AlterConfigOp> ops,
final IncrementalAlterConfigsOptions options);
If a configuration key is specified multiple times, that is an error, and an error code will be returned for each instance of the configuration key.
Similar to AlterConfigsResult, IncrementalAlterConfigsResult will have a values function which allows callers to determine the result of a specific configuration resource modifications, and an all function which will throw an exception if any operation failed.
@InterfaceStability.Evolving
public class IncrementalAlterConfigsResult {
private final Map<ConfigResource, KafkaFuture<Void>> futures;
ModifyConfigsResult(Map<ConfigResource, KafkaFuture<Void>> futures) {
this.futures = futures;
}
/**
* Return a map from resources to futures which can be used to check the status of the operation on each resource.
*/
public Map<ConfigResource, KafkaFuture<Void>> values() {
return futures;
}
/**
* Return a future which succeeds only if all the alter configs operations succeed.
*/
public KafkaFuture<Void> all() {
return KafkaFuture.allOf(futures.values().toArray(new KafkaFuture[0]));
}
}
Similar to AlterConfigsOptions, IncrementalAlterConfigsOptions will include a timeout value and a dry-run flag.
@InterfaceStability.Evolving
public class IncrementalAlterConfigsOptions extends AbstractOptions<IncrementalAlterConfigsOptions> {
private boolean validateOnly = false;
/**
* Set the request timeout in milliseconds for this operation or {@code null} if the default request timeout for the
* AdminClient should be used.
*/
public IncrementalAlterConfigsOptions timeoutMs(Integer timeoutMs) {
this.timeoutMs = timeoutMs;
return this;
}
/**
* Return true if the request should be validated without altering the configs.
*/
public boolean shouldValidateOnly() {
return validateOnly;
}
/**
* Set to true if the request should be validated without altering the configs.
*/
public IncrementalAlterConfigsOptions validateOnly(boolean validateOnly) {
this.validateOnly = validateOnly;
return this;
}
}
Protocol APIs
There will be a new IncrementalAlterConfigsRequest.
IncrementalAlterConfigsOp => INT8 0: SET 1: REMOVE 2: APPEND 3: SUBTRACT IncrementalAlterConfigsRequest (Version: 0) => [resources] validate_only validate_only => BOOLEAN resources => resource_type resource_name [configs] resource_type => INT8 resource_name => STRING configs => config_name config_op config_value config_name => STRING config_op => INT8 config_value => NULLABLE_STRING
The IncrementalAlterConfigsResponse is the same as the AlterConfigsResponse.
ModifyConfigsResponse (Version: 0) => [responses] responses => resource_type resource_name error_code error_message resource_type => INT8 resource_name => STRING error_code => INT16 error_message => NULLABLE_STRING
Compatibility, Deprecation, and Migration Plan
This change is backwards compatible because the existing alterConfigs RPC is retained. Clients will migrate to the new modifyConfigs RPC as needed.
Rejected Alternatives
We could have changed alterConfigs so that it had an incremental mode. This would have avoided creating a new RPC. However, in order to avoid breaking compatibility, the incremental mode could not have been made the default for AdminClient. We would also not have been able to deprecate the non-incremental mode. This would create a confusing and dangerous stumbling block for new users. Because the problems with non-incremental mode are not immediately obvious, it is likely that many users would have made the wrong decision about what API to use.