...
ApiVersions Request/Response Handling
The client does not know what which ApiVersions versions the brokers support broker supports as the ApiVersions is used for this purpose. Today, the client sends an ApiVersionsRequest (AVR) with the latest schema it is aware of. The broker handles it with the correct version if it knows it or sends back an ApiVersionsResponse v0 with an `UNSUPPORTED_VERSION` error to the client if it doesn't. When the client receives such error, it retries the whole process with the ApiVersionsRequest v0. It means that the broker won't get any additional information about the client if a newer version is used that it does not know about. To circumvent this, we propose to add the supported version of the ApiVersionsRequest in the response sent back to the client alongside the error. The client will be able to leverage this version to send back the correct ApiVersionsRequest instead of defaulting to version 0.
At the moment, the ApiVersionsRequest is handled in two different places in the broker: 1) in the SaslServerAuthenticator (when used); and 2) in the KafkaApis. Both places will be updated to ensure that all clients work. We have decided to not refactor the handling of the ApiVersionsRequest for now and it to leave it for further improvements.
...
A new separate request/response could be used for the purpose. This option has been discarded because it would add another round trip to the broker in the establishment of the KafkaChannel.
ApiVersionsRequest combined with "prefix-based" compatibility
We have considered removing the extra round-trip to the broker when the version of the AVR is unknown by ensuring that new fields would be added to the end of the ApiVersions Request and Response. This way, we could parse newer version of the request or the response with any previous version. We have discovered this solution because it would have obliged us to freeze the RequestHeader forever which is not wise.