...
Code Block |
---|
$ kafka-sentry -arg -r test -g test-group |
Granting privileges to a role
Here we grant some privileges to the role, test, so that users in
testGroup
can create a topic, testTopic
, and produce to it.
Allow users in test-group
to create a new topic from localhost.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Cluster=kafka-cluster->action=create" |
Allow users in test-group
to describe testTopic from localhost, which the user will create and use
.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=describe" |
Allow users in test-group
to write to testTopic
from localhost, this will allow the users to produce to testTopic
.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=write" |
Authorizable Resources
Authorizable resources are resources or entities in a Kafka cluster that require special permissions for a user to be able to perform some action on it. As of now Kafka has three authorizable resources.
- Cluster, this controls who can perform cluster level operations, like, creating a topic, deleting a topic, etc. This can only have one value
kafka-cluster
, as one Kafka cluster can not have more than one Cluster resources. - Topic, this controls who can perform topic level operations, like, producing to topic, consuming from topic, etc. Its value must match exactly with the topic name in Kafka cluster.
- Consumergroup, this controls who can perfrom consumergroup level operations, like, join an existing consumergroup, querying offset for a partition, describe a consumergroup, etc. Its value must exactly match
group.id
of a consumer group. - Host, this controls from where specific operations can be performed. It can be though of as a way to achieve IP filtering in Kafka. This can have a wildcard,
*
, as a value, which represents all hosts.
Authorized Actions
Each resource can have multiple actions that users can perform on them. Following operations are supported in Kafka, however not all actions are valid on all resources.
- ALL, this is a wildcard action, and represents all possible actions on a resource.
- read
- write
- create
- delete
- alter
- describe
- clusteraction
Authorizing Privileges
Privileges define what actions are allowed on a resource. A privilege is represented as a string in Sentry. Following are the criterias of a valid privilege.
- Can have at most one
Host
resource. If noHost
resource is specified in a privilege string,Host=*
is assumed. - Must have exactly one non
Host
resource. - Must have exactly one action specified at the end of privilege string.
Valid privilege strings
Host=*->Topic=test->action=ALL
Topic=test->action=ALL
Invalid privilege strings
Host=*->Host=127.0.0.1->Topic=test->action=ALL
, has multipleHost
resourcesCluster=kafka-cluster->Topic=test->action=ALL
, has multiple nonHost
resourcesTopic=test->action=ALL->action=read
, has multiple actionsCluster=cluster1->Topic=test->action=ALL
, should only havekafka-cluster
asCluster
valueaction=ALL->Topic=test
, action must be specified at the end of privilege string
Granting privileges to a role
Here we grant some privileges to the role, test, so that users in
testGroup
can create a topic, testTopic
, and produce to it.
Allow users in test-group
to create a new topic from localhost.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Cluster=kafka-cluster->action=create" |
Allow users in test-group
to describe testTopic from localhost, which the user will create and use
.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=describe" |
Allow users in test-group
to write to testTopic
from localhost, this will allow the users to produce to testTopic
.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=write" |
Create testTopic
.
Code Block |
---|
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic
$ kafka-topics.sh --list --zookeeper localhost:2181
testTopic |
Produce to testTopic
. Note that you will have to pass a config file, producer.properties, with information on jaas conf and other kerberos authentication related information. Here is more information.
Code Block |
---|
$ kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic --producer.config producer.properties
This is a message
This is another message |
Allow users in test-group
to describe a consumer group, testconsumergroup
, that it will be starting or joiningCreate testTopic
.
Code Block |
---|
$ kafka-topics.shsentry --creategpr --zookeeper localhost:2181r test --replication-factor 1 --partitions 1 --topic testTopic $ kafka-topics.sh --list --zookeeper localhost:2181 testTopicp "Host=127.0.0.1->Consumergroup=testconsumergroup->action=describe" |
Allow users in test-group
to read from a consumer group, testconsumergroup
, that it will be starting or joiningProduce to testTopic
. Note that you will have to pass a config file, producer.properties, with information on jaas conf and other kerberos authentication related information. Here is more information.
Code Block |
---|
$ kafka-console-producer.shsentry -gpr --broker-list localhost:9092 --topic testTopic --producer.config producer.properties This is a message This is another messager test -p "Host=127.0.0.1->Topic=testTopic->action=read" |
Allow users in test-group
to describe a consumer group, testconsumergroup
, that it will be starting or joining.read from testTopic
from localhost, this will allow the users to consumer from testTopic
.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Consumergroup>Topic=testconsumergrouptestTopic->action=describeread" |
Allow users in test-group
to read from a consumer group, testconsumergroup
, that it will be starting or joiningConsume from testTopic
. Note that you will have to pass a config file, consumer.properties, with information on jaas conf and other kerberos authentication related information. The config file must also specify group.id
as testconsumergroup
. Here is more information.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=read" |
Allow users in test-group
to read from testTopic
from localhost, this will allow the users to consumer from testTopic
.
Code Block |
---|
$ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=read" |
Consume from testTopic
. Note that you will have to pass a config file, consumer.properties, with information on jaas conf and other kerberos authentication related information. The config file must also specify group.id
as testconsumergroup
. Here is more information.
Code Block |
---|
$ kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopic --from-beginning --consumer.config consumer.properties
This is a message
This is another message |
Performance Comparison
...
-console-consumer.sh --zookeeper localhost:2181 --topic testTopic --from-beginning --consumer.config consumer.properties
This is a message
This is another message |
Performance Comparison
Following numbers are obtained by running Kafka's ProducerPerformance tests on a three node Kafka cluster, one node Zookeeper and one node Sentry service. All tests are run from same host and test topic has three partitions with three set as replication factor.
Future Work
Though Kafka authorization via Sentry works and it works nicely, there are a few more things that can be done here.
- Add caching of Sentry privilege objects, re-creating these privilege objects is costly and has been raised as concern on Sentry's dev mailing list as well. This will further improve Sentry's Kafka authorizer's performance.
- Add a native script, similar to the example added above, to allow users to get started with performing CRUD operations on roles and privileges out-of-the-box.
- Refactor the authorizer code after KIP-50 goes in Kafka.