Current state: "Accepted"
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Kafka authorizes access to resources like topics, consumer groups etc. by way of ACLs. The current supported semantic of resource name in ACL definition is either full resource name or special wildcard '*', which matches everything.
Kafka should support a way of defining bulk ACLs instead of specifying individual ACLs.
Example use cases:
This support will greatly simplify ACL operational story in a multi-tenant environment.
The examples below should help demonstrate the purposed functionality:
Each call retrieves ACLs stored in one path in ZK:adminClient.describeAcls(... TOPIC, "foobar" ...)
-> would return only ACLs from '/kafka-acls/foobar' pathadminClient.describeAcls(... TOPIC, "*" ...)
-> would return only ACLs from '/kafka-acls/*' path
Legacy constructors default resourceNameType to Literal and maintains existing contract:adminClient.describeAcls(... TOPIC, "foobar" ...)
-> still returns only ACLs from '/kafka-acls/foobar' pathadminClient.describeAcls(... TOPIC, "*" ...)
-> still returns only ACLs from '/kafka-acls/*' path
The user needs to know, up front, which prefixed resource paths exist to be able to query them:adminClient.describeAcls(... TOPIC, "foobar", Literal ...)
-> will return only ACLs from '/kafka-acls/foobar' pathadminClient.describeAcls(... TOPIC, "*", Literal ...)
-> will return only ACLs from '/kafka-acls/*' pathadminClient.describeAcls(... TOPIC, "foo", Prefixed ...)
-> will return only ACLs from '/kafka-prefixed-acls/foo' path
This allows user to discover all the ACLs affecting a specific resource:adminClient.describeAcls(... TOPIC, "foobar", Any ...)
-> will return all ACLs affecting topic "foobar", including any prefixed and wildcard ACLs.
The return value from describeAcls
will contain the resourceNameType field for each ACL, so the user can determine if it is literal or prefixed.
Acls on prefixed resource paths are never returned to earlier clients. Nor can older clients delete ACLs on prefixed resource paths.
Solution
The proposal is to enhance the SimpleAclAuthorizer to support prefixed ACLs.
This means that it will be possible to create ACLs of type: User:clientA has READ access on topic prefixed with 'orgA' from 'hostA', i.e clientA has READ access to all topics that start with `orgA` from hostA.
The concept of prefixed ACLs will be applicable only to resource names.
Storage model
Currently, ACLs are stored on ZK under path /kafka-acl/<resource-type>/<resource-name>.
For example:
ACLs for topic topicName will be stored under /kafka-acl/Topic/topicName.
ACLs for consumer group groupId will be stored under /kafka-acl/Group/groupId.
An example ACL definition looks like:
$ get /kafka-acl/Topic/topicName
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"},{"principal":"User:clientA","permissionType":"Allow","operation":"Write","host":"*"},{"principal":"clientB","permissionType":"Allow","operation":"Write","host":"host1"}]}
Current supported resource names are either full resource names like topicName or a special wildcard '*'.
$ get /kafka-acl/Topic/*
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"}]}
which means that clientA has read access to all topics from all hosts.
The challenge here is that there can be both literal and prefixed resource paths with the same name. Some resources like consumer groups don't have any defined naming convention, so can include any characters in their name. It is therefore not possible to prepend/append a 'special character' to prefixed names to distinguish them from literal ones.
We extend the same storage model to store prefixed ACLs in a different location 'kafka-prefixed-acl'. Changes will first be stored at 'kafka-prefixed-acl-changes'. Literal resources, including the wildcard resource '*', will continue to be stored in their original location.
$ get /kafka-prefixed-acl/Topic/teamA
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"}]}
ACLs write path
Write to a new location 'kafka-prefixed-acl'.
$ get /kafka-prefixed-acl/Topic/orgName
{"version":1,"acls":[{"principal":"User:clientA","permissionType":"Allow","operation":"Read","host":"*"}]}
ACLs read path
On read path, we look for all matching ACLs when authorize() is called.
Access will be allowed if there is at least one ALLOW matching acl and no DENY matching ACL (current behaviour is maintained). Note that the length of the prefix doesn't play any role here.
On downgrade, any prefixed ACLs will be ignored because they are in separate path. This means that any prefixed ACLs will be treated as if they were never added. This is fine for ALLOW ACLs, but might have security implications if DENY ACLs are ignored.
Constructors for Resource and ResourceFilter that don't take a ResourceNameType will be deprecated in favour of those that do.
Care has been taken to ensure legacy clients can neither add, list or remove prefixed ACLs. Clients wishing to use prefixed ACLs will need to upgrade their (admin)clients.