...

Discussion thread: 

JIRA: 

Motivation

Precedent

The KIP which established dynamic broker configuration, KIP-226, specified that this configuration data would be stored in ZooKeeper. It also established an encryption mechanism for secrets such as passwords. We would like to have the same level of protection for secret data in the post-ZooKeeper world of KRaft.

Threat Model

The threat model for the KIP-226 encryption was to protect against an attacker who had compromised the ZooKeeper service, but not the Kafka service. Since there is no more ZooKeeper service when using KRaft, our model is different. Our threat model is to protect against an attacker who has gained access to the metadata log, but not other aspects of the system (such as the system used to store encryption keys).

Different users may wish to store metadata encryption keys in different ways. For example, in a system with both local storage and remote network-accessed storage, it may make sense to store the metadata encryption keys on the local storage, but the metadata log on remote-accessed storage. Alternately, some users may want to store the metadata encryption key remotely, so that it doesn't appear on disk storage accessible to the broker at any time. A third user may not want to encrypt metadata at all. Therefore, this KIP makes the metadata encryption key mechanism both pluggable and optional.

Key Rotation

Finally, metadata key rotation is very important to security. We want it to be possible to do this rotation online, without shutting down the cluster and taking downtime. It should also be possible to do this without doing a rolling restart.

...

As mentioned previously, only secrets in the metadata log will be encrypted by this mechanism. That includes things like configurations which are passwords, and new metadata encryption keys themselves.

Metadata Encryption Keys

Overview

Metadata encryption keys are sequences of bytes used to encrypt the metadata via a symmetrical encryption algorithm. Each key is identified by a 16-byte UUID.

Registration

Metadata keys may be "registered," which associates them with a MetadataEncryptionKeyAccessor class.

Populated

A registered metadata key is "populated" when the key data is present.

Activated

A registered and populated metadata key may be "activated," which means that it becomes the key which is actually used for metadata encryption. Only one key can be active at once.

In order to facilitate key rotation, many metadata encryption keys may be "registered" with the system. However, only one key can be "active." The active key is the one which is actually used for encryption.

Each metadata encryption key is identified by a 16-byte UUID.This KIP introduces the concept of a metadata encryptor. Each encryptor is identified by a an unique 16-byte UUID and is capable of encrypting arbitrary binary data.

While a node can have any number of encryptors configured, only one encryptor is active at once. KRaft configuration records that contain secret data (such as passwords) are encrypted using the currently active encryptor.

...

.

Registered
    |
    |
    V

Mechanism

In general, 

Public Interfaces

...

This record registers a metadata encryption key , which is identified by a 16-byte UUID. The "mechanism" field is the full name of a Java class which implements the given metadata encryption key. Note that this record does not contain the actual encryption key. It simply registers it.in the metadata log. This record appears once in the metadata log and for each registered encryption key. It also appears once in the metadata snapshot image for each currently registered metadata encryption key.

{
  "apiKey": ...,
  "type": "metadata",
  "name": "RegisterMetadataEncryptionKeyRecordMetadataEncryptionKeyRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "KeyId", "type": "uuid", "versions": "0+",
      "about": "The metadata encryption key ID." },
    { "name": "AccessorClass", "type": "string", "versions": "0+",
      "about": "The MetadataEncryptionKeyAccessor class to use for this key." }
  ]
}

...

TransmitMetadataEncryptionKeyRecord

This record unregisters transmits a metadata encryption key in the metadata log. This record appears once in the metadata log for each registered encryption key that needs to be deregisteredtransmitted. It does not appear also appears once in the metadata snapshotimage for each currently registered metadata encryption key.

{
  "apiKey": ...,
  "type": "metadata",
  "name": "UnregisterMetadataEncryptionKeyRecordMetadataEncryptionKeyRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "KeyId", "type": "uuid", "versions": "0+",
      "about": "The metadata encryption key ID." },
    { "name": "AccessorClass", "type": "string",  ]
}

TransmitMetadataEncryptionKeyRecord

This record transmits a metadata encryption key in the metadata log. Note that the key is encrypted with the currently active encryption key.

"versions": "0+",
      "about": "The MetadataEncryptionKeyAccessor class to use for this key." }
  ]
}

ActivateMetadataEncryptionKeyRecord

MetadataEncryptionKeyReferenceRecord

This record appears in the metadata image This record appears once in the metadata log for each registered encryption key that needs to be transmitted. It does not appear in the metadata snapshot.

{
  "apiKey": ...,
  "type": "metadata",
  "name": "TransmitMetadataEncryptionKeyRecordRegisterMetadataEncryptionKeyRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "KeyId", "type": "uuid", "versions": "0+",
      "about": "The metadata encryption key ID." },
    { "name": "KeyDataAccessorClass", "type": "bytesstring", "versions": "0+",
      "about": "The MetadataEncryptionKeyAccessor class to use for this key." }
  ]
}

RegisterMetadataEncryptionKeyRecord

This record registers a metadata encryption key, which is identified by a 16-byte UUID. The "mechanism" field is the full name of a Java class which implements the given metadata encryption key. Note that this record does not contain the actual encryption key. It simply registers it.

This record appears once in the metadata log and once in the metadata snapshot for each registered encryption key.nregisterMetadataEncryptionKeyRecord

This record unregisters a metadata encryption key.

This record appears once in the metadata log for each registered encryption key that needs to be deregistered. It does not appear in the metadata snapshot.

{
  "apiKey": ...,
  "type": "metadata",
  "name": "UnregisterMetadataEncryptionKeyRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "KeyId", "type": "uuid", "versions": "0+",
      "about": "The metadata encryption key ID data. Encrypted with the current encryption key." }
  ]
}

...

ctivateMetadataEncryptionKeyRecord

This record activates a metadata encryption key in the metadata log.

...