Current state: (4x+1 binding, 1x+1 non-binding)
Discussion thread: here
Vote thread: here
JIRA:
Pull Request: here
Tools kafka-console-producer.sh
and kafka-console-consumer.sh
have the multiple option --property
to setup the MessageReader
for Producer (defined by --line-reader
) or the MessageFormatter
for Consumer (defined by --formatter
).
To setup the KafkaProducer
/KafkaConsumer
they have the multiple option --producer-property
/--consumer-property
and the config file option --consumer.config
/--producer.config
.
The idea is to offer the same possibility to use a config file option to setup the MessageReader
/MessageFormatter
in addition to the existing option --property
.
It will be useful in different use cases:
--property
(the default reader can have up to 11 properties)schema.registry.ssl.truststrore.password
for kafka-avro-console-producer
) (could also be mitigated by using env vars)Add an option:
--reader-config
to tool kafka-console-producer.sh
--formatter-config
to tool kafka-console-consumer.sh
As for --producer-property
/--consumer-property
with --consumer.config
/--producer.config
, any value from option --property
would override value from config file.
Before (multiple --property
options):
kafka-console-producer.sh [...] --property print.timestamp=true \ --property print.key=true \ --property print.offset=true \ --property print.partition=true \ --property print.headers=true \ --property print.value=true \ --property key.separator=':' \ --property line.separator=';' \ --property headers.separator='|' |
After (all properties in one file):
print.timestamp=true print.key=true print.offset=true print.partition=true print.headers=true print.value=true key.separator=: line.separator=; headers.separator=| |
kafka-console-producer.sh [...] --reader-config reader-config.properties |
When a property is in the config file and as option, option wins.
print.timestamp=true print.key=true print.offset=true print.partition=true print.headers=true print.value=true key.separator=: line.separator=; headers.separator=| |
kafka-console-producer.sh [...] --reader-config reader-config.properties --property key.separator='/' |
Then, the MessageReader
will get key.separator=/
, not key.separator=:
Implement the new options as described in Public Interfaces section:
ConsoleProducer
: new option "reader-config
" with description "Config properties file for the message reader. Note that --property takes precedence over this config.
"ConsoleConsumer
: new option "formatter-config
" with description "Config properties file to initialize the message formatter. Note that --property takes precedence over this config.
"Already implemented in pull request #12175.
No impact on existing users, just a new possibility available.
Launch the tools with the new config file option and check that properties from config file are applied to the reader/formatter.
Define a property in config file and in options and check that option override config file.
--config
to avoid confusion in the purpose of this config file (but on the other hand, the current --property
is quite generic already, maybe another KIP to change it).--read.config
/--formatter.config
, with ".
" as separator, as it's not consitent with the commonly used option --command-config