Child pages
  • KIP-145 - Expose Record Headers in Kafka Connect

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • StringConverter
  • ByteArrayConverter
  • JsonConverter

A new HeaderConverter implementation will be added to convert all built-in primitives, arrays, maps, and structs to and from string representations. Unlike the StringConverter that uses the toString() methods, the SimpleHeaderConverter uses JSON-like representation for primitives, arrays, maps, and structs, except for simple string values that are unquoted. This form corresponds directly to what many developers would think to serialize values as strings, and it allos the SimpleHeaderConverter to parse these any and all such values and most of the time to infer the proper schema. As such, this will be used for the default HeaderConverter used in the Connect worker.

The following table describes how these values will be persisted by the SimpleHeaderConverter.

Schema.TypeDescription of string represesntationExample
BOOLEANEither the true or false literals strings 
BYTE_ARRAYUTF-8 encoded string representation of the byte array 
INT8The string representation of a Java byte. 
INT16The string representation of a Java short. 
INT32The string representation of a Java int. 
INT64The string representation of a Java long. 
FLOAT32The string representation of a Java float. 
FLOAT64The string representation of a Java double. 
STRINGThe UTF-8 representation of the string, without surrounding quotes. 
ARRAYA JSON-like representation of the array. Array values can be of any type, including primitives and non-primitives. Embedded strings are quoted. However, when parsing, the schema will only be inferred when all values are of the same type. 
MAPA JSON-like representation of the map. Although most properly-created maps will have the same type of key and value, maps with any keys and values are also supported. Map values can be of any type, including primitives and non-primitives. Embedded strings are quoted. However, when parsing, the schema will be inferred only when the keys are strings and all values have the same type.
{ "foo": "value", "bar": "strValue", "baz" : "other" }
STRUCTA JSON-like representation of the struct. Struct object can be serialized, but when deserialized will always be parsed as maps since the schema is not included in the serialized form.
{ "foo": true, "bar": "strValue", "baz" : 1234 }
DECIMALThe string representation of the corresponding java.math.BigDecimal. 
TIMEThe IOS-8601 representation of the time, in the format "HH:mm:ss.SSS'Z'".16:31:05.387UTC
DATEThe ISO-8601 representation of the date, in the format "YYYY-MM-DD".2017-05-21
TIMESTAMPThe ISO-8601 representation of the timestamp, in the format "YYYY-MM-DD'T'HH:mm:ss.SSS'Z'".2017-05-21T16:31:05.387UTC

 

Configuration Properties

The Connect workers need to be configured to use a HeaderConverter implementation, and so one additional worker configuration named header.converter will be defined, defaulting to the StringConverterSimpleHeaderConverter. A similar configuration property with the same name and default will be added to the connector configuration, allowing connectors to override the worker's header converter. Note that each Connector task will have its own header converter instance, just like the key and value converters.

...